Rejected over RAG (so I built one)

The interview was going well. Agent architecture, safety boundaries, human-in-the-loop design: comfortable territory, I had built all of it. Then we started talking about RAG. I said the honest thing, which was also the fatal thing: I had designed retrieval systems on paper, but never run one in production. They chose another candidate.

Fair call. A design you have not operated is a guess with a diagram.

So I spent the next week building the design for real: embeddings, a vector index in Postgres, hybrid search, a reranker, and an eval harness. I built it inside my own project, where mistakes cost nothing. I expected to learn about components. Instead I learned one thing, six times, with a different bruise each time:

Every RAG mistake I made was a data question I answered too late. Not a model question. Not a framework question. A data question. The corpus and the queries decide the architecture. The components only carry out that decision. This article makes the case for checking your data first, with the numbers I paid to learn it.

Look at the data before you pick anything

The exact setup does not matter much. That is the point. I had a small corpus of short, expert-written knowledge units. An assistant retrieved them to ground its answers. The questions came from non-technical users. Your version might be support articles, policy documents, or runbooks. The lessons transfer because they depend on only two properties of the data. One hour of actually reading the corpus surfaced both, and both later dictated component choices.

The corpus was already atomic. Every unit was written by a person and complete on its own. There was no long document to split. So the entire chunking topic (sizes, overlaps, parent-document tricks) did not apply to this data. That is not skipping the chunking question. That is answering it. The first decision in RAG is: what is the atomic unit of retrieval? The answer comes from reading the corpus, not from a framework default. With long reports instead, the answer flips: embed small pieces, return the larger section around a hit. Same question, opposite answer, decided entirely by data structure.

The queries and the corpus speak different languages. Users ask “how much money did we make” and “which warehouse moves the most product”. The corpus answers in revenue, distribution centers, and sessions. Almost no shared words. Written down early, this one observation predicts most of what my experiments later confirmed. I still had to get burned twice before I trusted it.

Everything below is what happened where I checked too late, or checked the wrong thing.

Vectors are data too, so check where they came from

First burn. My eval harness could query any index version. On its first run against a second embedding model’s vectors, hit@1 fell from 93% to 14%. No error anywhere.

The bug: the question was embedded with model A, but searched against vectors from model B. Embeddings from different models live in unrelated coordinate systems. Similarity between them is not “worse”. It is meaningless. And nothing warns you, because the dimensions match and the queries run.

The lesson: an embedding is not a technical byproduct. It is data with an origin, and the origin must be enforced by the system, not remembered by people. Every vector row stores its model and dimensions. The query embedding model is derived from the index version being searched, never from global config. Mixed versions become structurally impossible. If your index rows do not record where their vectors came from, you do not have an index. You have a future incident.

Fit the retriever to your queries, not to the tutorial

Every tutorial says: combine dense search (meaning) with lexical search (keywords), merge the ranked lists, profit. My first test said: dense alone 93% hit@1, hybrid 89%. The extra leg made results worse. Nobody who wrote those tutorials had met my queries.

The cause was the language gap from the audit. My users paraphrase. Their questions share no tokens with the corpus, so the keyword leg returned nothing for 16 of 18 paraphrase queries. And when it did match, it sometimes matched garbage. For “average order value by traffic source” it confidently returned a completely wrong document, only because that document’s body contained the exact tokens avg_order_value and traffic_source. Rank fusion has no concept of confidence. It merges noise as happily as signal.

One layer deeper, still a data question: my “BM25” was not BM25. Postgres ts_rank has no inverse document frequency and no term saturation. A match on a common word scores the same as a match on a rare one. That is half of how the poisoning happened. Check what your infrastructure actually computes over your text, not what the blog post assumed.

Then the confirmation from the other side. When I later grew the eval set and added more domain-vocabulary questions, hybrid’s penalty disappeared (86 vs 87), and on its home class it scored 95%. Same components, different query mix, opposite verdict. Hybrid is not good or bad. It is a bet on your query distribution, and only your data can price that bet. If real queries carry exact tokens (IDs, SKUs, error codes), buy the insurance: dense retrieval cannot see E-1147 at all, because the tokenizer cuts it into meaningless pieces. If your users paraphrase, measure before buying.

”Better model” is not a property of the model

A newer embedding model shipped in the middle of my project. The upgrade instinct is fashion. The data-first move is a fitting: build the new model’s vectors next to the live ones (the index is keyed by version), run the same golden set against both, and let the corpus vote.

It voted no: 86% against 93% for the old model, on my data, at my dimensionality. The old model stayed active. The rejection is documented in the eval report.

Then the finding that paid for the whole exercise. After I grew the eval set to 122 questions, the newer model still lost as a single-stage ranker (80 vs 87). But with a reranker on top it won, 97 to 91. Its candidate lists were slightly richer, and the precision stage used exactly that. The same two models rank in opposite order depending on what sits above them.

So “which embedding model is better” is an incomplete question. The complete question is: better inside which pipeline, on which data. Components do not have quality. Configurations do. A leaderboard cannot answer that. Only your corpus, your queries, and an eval built from them can.

The reranker follows the same rule: it is a response to a measurement, not a default. High hit@10 with weak hit@1 means ordering is the problem, and a precision stage earns its cost. Mine added a full second per question, double my retrieval budget. If your top results are already right, skip the reranker and spend that budget on injecting a little more context instead. Precision stages fix ordering. They fix nothing else.

The eval is how the data speaks, so keep it able to speak

How does the data cast all these votes? Through a golden set: realistic questions mapped to acceptable documents, split by query type (paraphrase, exact-token, deliberately vague), scored with hit@k and MRR. The eval turns “I think” into “on my data”.

Instruments wear out too. At 28 questions, one configuration scored a perfect 100%. The correct reaction to a perfect score is suspicion: at that size, the top two configs differ by one single query. A saturated eval stops distinguishing configs exactly where your decisions live. So I grew the set to 122 questions: two fresh paraphrases per document, more domain vocabulary, more vague ones. The ceiling behaved as theory promised:

config	hit@1 (n=28)	hit@1 (n=122)
old model, dense	93%	87%
old model, hybrid+rerank	96%	91%
new model, dense	86%	80%
new model, hybrid+rerank	100%	97%

The perfect score died, as perfect scores should. And the bigger eval is what resolved the model flip above, which the smaller eval had blurred into a coin toss.

Two rules worth memorizing. First: an eval you cannot fail is a decoration. Second, the diagnostic: high hit@k with low MRR means retrieval finds the right document but ranks it badly (add a precision stage); low hit@k means the right document never arrives (no reranker can save you; fix recall or fix the corpus).

The class breakdown added one honest note. Deliberately vague questions (“how is the website doing?”) score 62–69% hit@1 in every configuration. Some questions are ambiguous because they are ambiguous. The fix is a clarifying question in the product, not a better index. Hearing that difference is also part of checking your data.

A dataset is a living thing, so keep checking it

A corpus that fits today drifts tomorrow. Most corpora contain checkable claims: code that should run, links that should resolve, names and values that should still exist in the systems they reference. That gives drift two forms. I staged both against my own corpus.

Structural drift makes a document loudly wrong: something it references stops existing. A scheduled validator re-checks every document’s claims against reality. It caught my staged break at once and quarantined the document. Then came the humbling part: the serving index kept serving the condemned document, because index loading only inserted and updated rows. Nothing ever deleted removed ones. The source of record said “quarantined”. Production said “great, shipping it”. Deletes are part of index sync, not an optimization. Curation that does not reach serving is just for show.

Semantic drift makes a document silently wrong: everything still runs and resolves, but the meaning no longer matches reality. My staged version was a query filtering on a value that no longer occurs in the data. It runs perfectly and returns a confident zero. No layer complains, because no layer knows what the content means. This class needs meaning-aware checks: validate the claims inside documents against the live world, track your metrics over time and alert on sudden jumps, watch the empty-result rate in production. Structural checks are nearly free. Nothing validates truth for free.

The same principle guards the corpus entrance. New documents pass a curation funnel that ends with a human, because one plausible but wrong document poisons every retrieval that touches it. A knowledge base stays golden only while someone checks the gold.

The decision order

The whole article in four lines, in the order I would defend on a whiteboard:

Read the data and the queries first. What is the atomic unit? What words do real questions use? Fifty real queries take twenty minutes and overrule any tutorial.
Dense is the default. Lexical is insurance you buy against a named, observed risk (exact tokens in real queries). And check what your lexical scorer actually computes.
Rerank in response to a measurement (good hit@10, weak ordering), never by default. Otherwise spend the latency budget on more context.
Discuss infrastructure last. By then the corpus, the queries, and the eval have usually made the choice already.

Data first. Then measurement. Architecture last, and reluctantly.

The cheat sheet

The compressed version, for the night before an interview.

Start with the data. The atomic unit of retrieval is decision number one, made by reading the corpus. Authored units: no chunking. Long documents: embed small pieces, return the larger section. Chunks that cannot stand alone: prepend context before embedding. Then read real queries and classify them. Paraphrase-heavy or exact-token-heavy decides the retriever family before any benchmark does.

Metrics. hit@k (also called success@k): was an acceptable document in the top k? This is the functional floor, since the model never sees anything below k. MRR@k: one divided by the rank of the first good document, averaged. It measures ordering. “Recall” strictly means the fraction of all relevant documents retrieved; with any-acceptable scoring, say hit rate. nDCG is only needed for graded relevance. The diagnostic: high hit@k plus low MRR, add a reranker; low hit@k, fix recall.

Dense vs lexical vs hybrid. Dense searches by meaning and is blind to exact tokens (IDs, SKUs, error codes). Lexical searches by tokens and is blind to paraphrase. BM25 is IDF times term saturation times length normalization; verify your “BM25” actually has the IDF. Hybrid merges both, usually with RRF (rank-based, no tuning, and no confidence, so it merges noise as happily as signal). Whether hybrid helps is a property of your query distribution. Measure.

Pipeline hygiene. Version every vector with model and dimensions. Derive the query-embedding model from the index version. Blue/green upgrades: build the new version alongside, evaluate both, flip a pointer, keep the old version for rollback. Index sync includes deletes, never just upserts. The source files are the system of record; the index is disposable.

Two-stage retrieval. Stage one buys recall (judge it by hit@N). Stage two buys precision (a cross-encoder or LLM reranker; expect roughly ten times the latency of the vector query). My stack: embedding round trip ~500ms, vector search ~1ms, rerank ~1s.

Evals. A golden set of questions mapped to acceptable documents, split by query class, because aggregates lie across classes. Run it on every change. A saturated eval means grow the set, not celebrate: mine went from 28 to 122 questions, pulled a 100% config down to 97%, and reversed a model comparison. Retrieval evals and answer evals are different layers. Debug them separately.

Corpus lifecycle. Validate continuously. Structural checks are cheap: does the code run, do the links resolve, do referenced things exist. Semantic checks need meaning: claims against the live world, metric baselines, empty-result alerts. Curation gates entry, and quarantine must propagate to serving.

The ending I would write for the interview

If they asked me today: RAG is mostly a search problem. The generation half is nearly solved; give a modern model the right context and it answers well. Every hard problem lives on the search side, and the search problem is a data problem before it is an algorithm problem. I know because I mixed vector spaces and watched 93 become 14. Because my corpus’s vocabulary gap predicted the hybrid verdict before the test confirmed it. Because the “worse” embedding model became the better one when the pipeline above it changed. Because my eval hit 100% and the right response was to make it harder until it could disagree with me again.

Check the dataset. Fit the solution to it. Everything else is fashion.

One week. One rejection, well spent.