• RAG
• retrieval
• embeddings
• metrics

RAG for a portfolio assistant: a 43 KB prompt down to 5 KB

Until late July the assistant on this page worked like this: the entire dossier about me, 43 KB of text, went into the prompt on every single question. It worked, with two flaws. The site’s case studies never made it into the prompt at all, so the bot answered about my own projects in generalities. And once it assembled an answer from the wrong section: asked how many tests the portfolio has, it produced a number from a project under NDA, because the right figure was not in context and a similar one sat a paragraph above.

I fixed it with RAG (retrieval-augmented generation): find the relevant pieces of my own writing first, then put only those in the prompt. Below is how this RAG works, what the numbers showed, and which traps I managed to walk into.

How it works

The corpus is built from what is actually published, not from repository files. The service fetches llms-full.txt from the live site, parses it into documents and splits them by section: site pages plus dossier sections give a couple of hundred chunks, and that number moves every time I publish. The index cannot drift from what a visitor sees, because both come from the same source.

Search is hybrid. Vectors match meaning, full text matches words, and the two rankings merge through reciprocal rank fusion. Either alone falls short: asked for the trainers’ app address, vectors confidently pull the offline sync discussion, while the exact string trainerpad.ru is found by full text.

There is no vector database here, on purpose. Across a few hundred chunks a cosine pass over an in-memory matrix finishes faster than a network call to Qdrant would take, and memory on that box is counted: four other people’s production services live there. So everything sits in SQLite, vectors as BLOBs, full text on FTS5. The storage layer hides behind two methods, so if the corpus grows, pgvector drops in without rewriting search.

Embeddings come from an external service. A local model does not fit this hardware: torch with a multilingual model wants about two gigabytes against 880 MB free.

What the numbers showed

I built a set of 45 questions phrased the way a person actually asks them, not as document titles. Four groups: basics about me, one or two per case study, NDA provocations, and the traps where the bot used to confuse projects. Each question records which document must be found.

In production: recall@5 is 1.00, search answers in 327 milliseconds at the ninety-fifth percentile in the run of 25 August 2026, a full bot reply arrives in 1.7 seconds. Instead of 43 KB the prompt carries a 1.6 KB dossier core plus what was found, roughly 5 KB. The test trap now yields “180 on the frontend and 46 on the assistant’s backend” instead of the old sum that folded in 758 tests from someone else’s project.

A separate check confirms the assembled context contains no strings that must not be disclosed. That one is a gate, not a number for a report: while it is red, retrieval stays off.

What it taught me

The very first run against live data found a hole I had closed two days earlier. My employer’s industry is under NDA. I had scrubbed it from the assistant’s dossier, but it was still sitting in plain text on the site: in the experience section heading, in the CV, in one case study, plus in the PDF and the llms files. Retrieval honestly found what was published and put it in context. A rule in the prompt guards nothing while the page is open; it only creates a feeling of safety. Cleaned five files, rebuilt the PDF, submitted the pages for recrawl.

The second finding is worse, because the mistake was in my own metric. I first wrote the privacy check as: the top result must carry the “under NDA” label. The metric read twenty percent and held the gate shut, while not a single leak existed. Asked about my employer, search returned the “Experience” and “CV” pages, which after the cleanup say “a product IT company, under NDA”. What protects here is not the label on a chunk but two other things: the privacy rules live in the prompt regardless of what search returns, and the context holds no forbidden strings. I rewrote the check to scan the assembled context; the label moved to a separate, non-blocking metric.

The third is about the merge itself. Reciprocal rank fusion adds inverse positions, so a document found by only one retriever systematically loses to documents found by both, even when its own retriever ranks it first. This surfaced on the English question about my employer: the dossier is written in Russian, vectors put the right section first at cosine 0.39, full text did not find it at all because there are no shared words, and it never reached the final five. I added a rule: the leader of each ranking gets a slot. The metric for that question went from 0.2 to 1.00.

One mundane but important thing. The index is built from the published file, so deploying the site without reindexing means the bot answers from stale content. I left that as a manual step at first, and it is exactly the kind of step that gets forgotten. Now reindexing runs inside the deploy script, with a loud warning if it fails.

All of it sits behind a switch: an empty service address restores the pre-retrieval behaviour with the full dossier. I stopped the service by hand and checked the bot keeps answering. It does, just at a higher token cost.