Debug RAG Retrieval: Why Your RAG Returns Wrong Answers

Your RAG system returns a confident, well-written, wrong answer. You already tried the obvious things — you tuned the chunk size, you bumped top-k from 5 to 10, maybe you swapped the embedding model. It got a little better, or it got worse, and you are not sure which change did what.

The problem is not that you lack fixes. Search “improve RAG accuracy” and you will drown in fixes: ten techniques, seven pitfalls, fifteen chunking strategies. The problem is that nobody tells you which fix is yours. So you apply them semi-randomly, and because each one takes an afternoon and a re-index, you burn a week finding out that contextual chunking did nothing because your bug was in the prompt all along.

This guide inverts that. We diagnose first, fix second. You will run a ten-minute test that tells you whether your bug lives in retrieval or generation, then measure the specific stage that is broken, then apply only the fix your evidence points to.

One warning before we start, because it shapes everything below. While researching this piece I tried to trace the most-repeated statistic in this entire subject area — that “70–73% of RAG failures are retrieval failures.” It does not have a source. It traces back through a chain of blogs to an uncited assertion, and it mutated along the way. I will come back to it, because the way that number spread is itself a lesson in how to debug RAG.

Why RAG fails quietly

A traditional bug throws. RAG returns a fluent paragraph that reads exactly like a correct answer and happens to be false. There is no stack trace, no exception, no red test. That is the whole difficulty: the failure is semantic, and your test suite has no idea.

It helps to remember what a RAG pipeline actually is. If you want the ground-up version, we covered it in RAG explained — but the short version is that a question travels through a series of stages, and each stage can silently destroy the answer:

question
   │
   ▼
[1] query processing ──── the question is embedded as-is,
   │                      even if it's vague or under-specified
   ▼
[2] retrieval ────────── vector search returns top-k chunks
   │                      (right chunk may never be reachable)
   ▼
[3] ranking ──────────── the right chunk is in the pool
   │                      but sits at position 14 of 20
   ▼
[4] context assembly ─── chunks are concatenated into a prompt
   │                      (right chunk + 4 confident distractors)
   ▼
[5] generation ───────── the model reads the context
   │                      and may still ignore or misread it
   ▼
answer

Every one of those stages produces plausible output when it fails. Retrieval always returns something — it returns the nearest neighbours, and if nothing relevant exists, the nearest neighbours are simply the least-irrelevant chunks in your corpus. Cosine similarity does not have a concept of “no good match.” It has a concept of “least bad match,” and it reports it with the same confidence.

So the first question is never “how do I make it better.” It is “which stage broke.”

The oracle test: retrieval or generation, in ten minutes

Here is the split that matters. Everything downstream depends on it, and it takes one script.

Take a question your system answers wrong. Find, by hand, the chunk in your corpus that actually contains the answer. Now run the question twice:

• Arm A — your real pipeline. Retrieve normally, build the prompt, generate.
• Arm B — the oracle. Skip retrieval entirely. Hand the model the known-good chunk and nothing else.

Then read the 2×2:

                    │  Arm B (oracle context)  │  Arm B (oracle context)
                    │  answer CORRECT          │  answer WRONG
────────────────────┼──────────────────────────┼─────────────────────────
Arm A (real)        │   pipeline works on      │        —
answer CORRECT      │   this query             │
────────────────────┼──────────────────────────┼─────────────────────────
Arm A (real)        │   RETRIEVAL is the bug   │  GENERATION is the bug
answer WRONG        │   (the model can use the │  (model fails even with
                    │   chunk; it never got it)│   the answer handed to it)

That bottom row is the whole game. If Arm B is correct and Arm A is wrong, your generator is fine and your retrieval is broken — stop tuning prompts. If Arm B is also wrong, retrieval is irrelevant to this failure; the model had the answer in front of it and still blew it, so the bug is in your prompt, your instructions, or the model itself.

There is a third case worth separating out, and it is the one most write-ups miss. Check whether the oracle chunk was in your retrieved set. If the right chunk was retrieved and the answer is still wrong, you do not have a retrieval bug in the usual sense — you have context dilution. The chunk was there; distractors drowned it. That is a different fix (reranking, lowering k, filtering) than “the chunk is unreachable” (chunking, embeddings, hybrid search).

Here is the whole diagnostic. No framework, no dependencies — swap call_llm for whatever provider you use:

"""Isolate retrieval failure from generation failure by force-feeding a known-good chunk."""
from dataclasses import dataclass

PROMPT = """Answer the question using ONLY the context below.
If the context does not contain the answer, reply exactly: INSUFFICIENT_CONTEXT

Context:
{context}

Question: {question}
Answer:"""


@dataclass
class Probe:
    question: str
    oracle_chunk: str        # the chunk you KNOW contains the answer
    expected_substring: str  # cheap deterministic grader


def build_prompt(question: str, chunks: list[str]) -> str:
    ctx = "\n\n---\n\n".join(f"[{i}] {c}" for i, c in enumerate(chunks, 1))
    return PROMPT.format(context=ctx, question=question)


def diagnose(probe: Probe, retriever, call_llm, k: int = 5) -> dict:
    retrieved = retriever(probe.question, k)

    # Arm A: the real pipeline
    ans_real = call_llm(build_prompt(probe.question, retrieved))

    # Arm B: oracle — force-feed the known-good chunk, bypassing retrieval
    ans_oracle = call_llm(build_prompt(probe.question, [probe.oracle_chunk]))

    ok_real = probe.expected_substring.lower() in ans_real.lower()
    ok_oracle = probe.expected_substring.lower() in ans_oracle.lower()

    # Did retrieval even surface the oracle chunk?
    oracle_retrieved = any(probe.oracle_chunk[:60] in c for c in retrieved)

    if ok_real:
        verdict = "PASS — pipeline works on this query"
    elif ok_oracle and not oracle_retrieved:
        verdict = "RETRIEVAL BUG — oracle chunk never retrieved; generator is fine"
    elif ok_oracle and oracle_retrieved:
        verdict = "CONTEXT DILUTION — chunk was retrieved but distractors drowned it"
    else:
        verdict = "GENERATION BUG — model fails even with the answer handed to it"

    return {
        "verdict": verdict,
        "oracle_retrieved": oracle_retrieved,
        "answer_real": ans_real,
        "answer_oracle": ans_oracle,
    }

Two details that make this work better than it looks.

The INSUFFICIENT_CONTEXT escape hatch in the prompt is not decoration. Without it, a model handed useless context will invent an answer rather than admit defeat, and you lose the signal you are testing for. With it, “INSUFFICIENT_CONTEXT” becomes a clean, greppable vote for “retrieval gave me nothing.”

The expected_substring grader is deliberately dumb. You do not need an LLM judge to check whether the string 30 days appears in an answer about a refund window. Save the judge for open-ended quality; use substring matching for factual probes and keep your diagnostic loop fast and free.

Run this over ten failing questions. The verdict distribution tells you where to spend the rest of your week.

The decision tree

Once the oracle test has sorted your failures, the path forks. Follow the branch your evidence actually landed on:

Answer is wrong
│
├─ Oracle context → answer CORRECT?
│  │
│  ├─ NO ──────────────────► GENERATION BRANCH
│  │                         The model had the answer and missed it.
│  │                         → prompt/instructions, model choice,
│  │                           conflicting priors, output format
│  │                         → STOP. Chunking will not help you.
│  │
│  └─ YES ─────────────────► RETRIEVAL BRANCH
│     │                      The model works. It never got the chunk.
│     │
│     ├─ Was the oracle chunk in the retrieved set?
│     │  │
│     │  ├─ YES ──────────► CONTEXT DILUTION
│     │  │                  It was there; distractors won.
│     │  │                  → rerank, lower k, filter, reorder
│     │  │
│     │  └─ NO ───────────► TRUE RETRIEVAL MISS
│     │     │
│     │     └─ Is the chunk in the top-50 candidate pool?
│     │        │
│     │        ├─ YES ────► RANKING PROBLEM
│     │        │            Reachable, ranked badly.
│     │        │            → cross-encoder reranker, hybrid search
│     │        │
│     │        └─ NO ─────► REPRESENTATION PROBLEM
│     │                     Not reachable at any k.
│     │                     → chunking, embedding model,
│     │                       hybrid/BM25, query rewriting
│     │                     → a reranker CANNOT save you here

That last distinction saves the most time. A reranker only reorders what retrieval already returned. If the right chunk is not in the candidate pool at k=50, a reranker is strictly incapable of helping, and every hour spent adding one is wasted. Check the pool before you buy the tool.

To check it, retrieve deeply and look for your oracle chunk:

# Is the right chunk reachable AT ALL, or just ranked badly?
deep = retriever(probe.question, k=50)
pos = next((i for i, c in enumerate(deep, 1)
            if probe.oracle_chunk[:60] in c), None)

if pos is None:
    print("REPRESENTATION problem — not in top-50. Reranking cannot help.")
elif pos > 5:
    print(f"RANKING problem — found at position {pos}. A reranker should fix this.")
else:
    print(f"Reachable at position {pos} — look at dilution/generation instead.")

Show the chunks you did not retrieve

Most RAG debugging UIs show you what came back. That is the wrong half.

Researchers at Pittsburgh and Berkeley ran a think-aloud study with 12 practitioners debugging RAG pipelines (RAG Without the Lag, CHI 2026) and found that developers check retrieval before they look at anything else — one participant put it as “I don’t wanna work on other sh*t until I know I can retrieve the right documents.” The paper’s other useful finding: 71.3% of the parameter changes participants wanted to make would have required a full re-index in a traditional workflow, which is exactly why people give up and start guessing instead.

The practical move is to always print the near-misses next to the hits:

def explain(question, retriever, oracle_chunk, k=10):
    hits = retriever(question, k, with_scores=True)
    print(f"Q: {question}\n")
    for i, (chunk, score) in enumerate(hits, 1):
        mark = "★" if oracle_chunk[:60] in chunk else " "
        print(f"{mark} {i:2d}. {score:.4f}  {chunk[:90]}")

When you look at that output, you are not asking “is the right chunk here.” You are asking why the wrong chunks beat it. Those near-misses are the actual diagnostic data. Usually one of a few stories is visible immediately:

• The top chunks all share vocabulary with the question but none answer it → your embeddings are matching topic, not intent.
• The right chunk is split mid-procedure → chunking boundary problem.
• The winners are all from one stale document → data quality, not retrieval.
• The right chunk scores 0.81 and a wrong one scores 0.83 → ranking, and a reranker is the fix.

Measuring retrieval properly

Once you know retrieval is the branch, stop eyeballing and measure. You need a ground-truth set: real queries, each labelled with the chunk(s) that should be retrieved.

Source these from your logs, not your imagination. This is the step people get wrong. If you ask an LLM to generate questions from your chunks, it will phrase them using the chunk’s own vocabulary, and retrieval will score beautifully because the match is nearly lexical. Real users do not use your documentation’s words. That vocabulary gap is the thing you are trying to measure, and synthetic queries engineer it away. Thirty real queries beat three hundred synthetic ones.

Then compute the metrics by hand. They are four lines each, and owning them means you know exactly what you measured:

import numpy as np


def precision_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    """Fraction of the top-k that are relevant. Denominator is k, not len(topk)."""
    if k <= 0:
        raise ValueError("k must be > 0")
    hits = sum(1 for d in retrieved[:k] if d in relevant)
    return hits / k


def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    """Fraction of all relevant docs found in the top-k."""
    if not relevant:
        return float("nan")  # undefined — must not silently count as 1.0
    hits = sum(1 for d in retrieved[:k] if d in relevant)
    return hits / len(relevant)


def reciprocal_rank(retrieved: list[str], relevant: set[str]) -> float:
    """1 / rank of the FIRST relevant doc (1-indexed). 0.0 if none found."""
    for i, d in enumerate(retrieved, start=1):
        if d in relevant:
            return 1.0 / i
    return 0.0


def mrr(runs: list[tuple[list[str], set[str]]]) -> float:
    """Mean reciprocal rank across queries."""
    if not runs:
        return float("nan")
    return float(np.mean([reciprocal_rank(r, rel) for r, rel in runs]))

Note the nan in recall_at_k. A query with no labelled relevant chunk has an undefined recall, not a perfect one. Returning 1.0 there — which the naive implementation does, since 0/0 gets special-cased away — silently inflates your average with every unlabelled query. It is a real bug and it is easy to ship.

If you use NDCG, there is a trap worth knowing. There are two incompatible gain conventions:

def dcg_at_k(gains, k: int, exponential: bool = True) -> float:
    """
    exponential=True  -> (2^rel - 1) / log2(i+1)   [TREC / Burges / Kaggle standard]
    exponential=False -> rel / log2(i+1)           [what sklearn.metrics.ndcg_score uses]

    The two agree ONLY for binary relevance (2^1-1 == 1, 2^0-1 == 0).
    """
    g = np.asarray(gains, dtype=float)[:k]
    if g.size == 0:
        return 0.0
    discounts = np.log2(np.arange(2, g.size + 2))  # log2(i+1), i 1-indexed
    numer = (np.power(2.0, g) - 1.0) if exponential else g
    return float(np.sum(numer / discounts))


def ndcg_at_k(retrieved, relevance: dict, k: int, exponential: bool = True) -> float:
    """`relevance` maps doc_id -> graded relevance (0 = irrelevant).
    The ideal ranking is computed over the FULL judged pool, not just what was retrieved."""
    gains = np.array([relevance.get(d, 0.0) for d in retrieved[:k]], dtype=float)
    dcg = dcg_at_k(gains, k, exponential)
    ideal = np.array(sorted(relevance.values(), reverse=True), dtype=float)
    idcg = dcg_at_k(ideal, k, exponential)
    return dcg / idcg if idcg else 0.0

sklearn.metrics.ndcg_score uses linear gain. TREC and most leaderboards use exponential. Under binary relevance they coincide exactly, so nobody notices — but the moment you move to graded relevance, your numbers and sklearn’s will quietly disagree, and you will spend an afternoon hunting a bug that is a convention mismatch.

Reading your own curve

Do not chase a number someone else published. Measure recall at several k values on your own set and read the shape:

What you see	What it means	Where to go
recall@20 high, recall@5 much lower	Chunk is reachable but ranked poorly	Reranker, hybrid search
recall@20 and recall@5 both low	Chunk is not reachable at any k	Chunking, embeddings, BM25, query rewriting
recall@5 high, answers still wrong	Retrieval is fine	Go to the generation branch
recall high, precision@5 low	Right chunk plus many distractors	Rerank, lower k, filter
Recall varies wildly across queries	One query class is broken	Segment by query type before averaging

That last row deserves emphasis. A mean hides the bug. If 80% of your queries retrieve perfectly and 20% fail completely, your average looks acceptable and one entire class of user question is silently broken. Always look at the distribution, and always read the failures individually. The single most useful debugging habit in RAG is opening the ten worst queries and reading what came back.

To put published numbers in perspective: one enterprise RAG case study (Raina & Gales, arXiv:2405.12363) reports standard RAG recall of 65.5% at top-1 rising to 89.3% at top-5. That gap between k=1 and k=5 is the shape you are looking for — it means ranking, not reachability, was the constraint. But note the scope before you treat those figures as targets: that is a single-dataset study on restructured SQuAD with all-mpnet-base-v2, and the authors themselves call it “a simple case study.” Those numbers are the baseline the paper exists to beat, not a benchmark for your system.

The generation branch

If the oracle test said generation, the retrieval fixes are all irrelevant. Here is what actually goes wrong when a model has the answer and misses it.

Distractors. The correct chunk arrived with company. Cuconasu et al. (SIGIR 2024, arXiv:2401.14887) found “a clear pattern of progressive accuracy degradation as the number of distracting documents increases,” and — the part worth internalising — “adding just one distracting document causes a sharp reduction in accuracy, with peaks of 0.24 (−25%).” One bad chunk. Note this was an oracle setup rather than a real top-k sweep, but the direction is unambiguous.

Conflicting priors. When retrieved context contradicts what the model already believes, it does not reliably defer to the context. The Stanford ClashEval study (arXiv:2404.10198) measured exactly this — how often a model abandons its own correct answer when handed wrong context:

Model	Adopts wrong context over correct prior
GPT-3.5	62.6%
GPT-4o	60.8%
Llama-3	52.9%
Gemini 1.5	49.0%
Claude Sonnet	40.1%
Claude Opus	31.3%

You will see this cited around the web as “models abandon their correct answer over 60% of the time.” That framing comes from the paper’s own abstract, so it is not invented — but look at the table. Only two of six models exceed 60%; the mean is closer to 49.5%, and the best model is at half the headline. These are also mid-2024 model versions. The honest takeaway is not a number, it is a behaviour: your model will often believe a bad chunk over its own correct knowledge, and how often depends heavily on which model you run. That is a reason to fix retrieval, not a reason to quote a statistic.

Position. The classic result here is Lost in the Middle (Liu et al., TACL 2023): a U-shaped curve where models use information at the beginning and end of the context far better than the middle, with GPT-3.5-Turbo dropping more than 20% in the worst case. Apply it with care — that study tested 2023-era models at roughly 4K tokens, which is a short-context result by today’s standards. Later work confirms long-context degradation is real (see NoLiMa, where 11 of 13 models claiming 128K+ fell below half their short-context baseline at 32K) but that measures degradation with length, not position. In 2026 the U-shape is pronounced in some models and largely absent in others. Test yours; don’t assume.

Practical generation-branch fixes, roughly in order of payoff:

1. Give the model an out. An explicit INSUFFICIENT_CONTEXT instruction converts silent fabrication into a detectable signal.
2. Cut k. If distractors are the problem, fewer chunks is a fix, not a regression.
3. Order deliberately. Put the highest-ranked chunk last (nearest the question) if your model shows recency bias.
4. Demand citations. Requiring the model to cite chunk IDs makes misgrounding visible — and lets you catch answers that cite a chunk that doesn’t support them.
5. Try a different model. Per ClashEval, context-adherence varies by a factor of two across models. This is a real lever.

Fixes for the retrieval branch

Only read this section if the oracle test sent you here.

Hybrid search: the highest-payoff fix for representation problems

Dense embeddings match meaning and are bad at exact tokens — error codes, product SKUs, function names, rare proper nouns. BM25 is the opposite. Combining them fixes a large fraction of “the chunk is unreachable” failures, because most such failures are a query with a rare literal in it.

Fuse with Reciprocal Rank Fusion, which needs no score normalisation — it only reads ranks:

def reciprocal_rank_fusion(ranked_lists, k: int = 60, weights=None):
    """
    RRF (Cormack et al., 2009):  score(d) = sum_over_lists  w_i / (k + rank_i(d))
    rank is 1-indexed. k=60 is the constant from the original paper.
    Docs missing from a list contribute nothing (no imputation).
    """
    weights = weights or [1.0] * len(ranked_lists)
    scores: dict[str, float] = {}
    for lst, w in zip(ranked_lists, weights):
        for rank, doc in enumerate(lst, start=1):
            scores[doc] = scores.get(doc, 0.0) + w / (k + rank)
    return sorted(scores.items(), key=lambda x: -x[1])

RRF’s appeal is that it sidesteps the scale problem entirely: cosine similarities and BM25 scores are not comparable quantities, and any attempt to normalise them into a weighted sum requires tuning you will get wrong. Ranks are ranks.

A sharp edge with rank_bm25: get_scores() returns a score for every document in the corpus, including zeros for documents sharing no terms with the query. Naively sorting and taking the top-k hands pure noise into your fusion at ranks 2 and 3, where RRF happily treats it as signal. Filter first:

scores = bm25.get_scores(tokenize(query))
ranked = [(i, s) for i, s in enumerate(scores) if s > 0]   # ← drop zero-score docs
ranked.sort(key=lambda x: -x[1])
bm25_list = [corpus_ids[i] for i, _ in ranked[:k]]

Also note that rank_bm25 does no tokenization of its own — whatever tokenize() you pass in is your lexical model. Lowercasing and splitting on whitespace will underperform on anything with punctuation-heavy identifiers. And the library has been unmaintained since early 2022; bm25s is a faster, actively maintained alternative if you are starting fresh.

Reranking: the fix for ranking problems

A cross-encoder reads the query and document together rather than embedding them separately, which is far more accurate and far slower. That trade-off dictates the architecture: retrieve broadly with a cheap bi-encoder, rerank the top ~50 with the expensive model.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")

candidates = retriever(query, k=50)          # cheap, broad
pairs = [(query, c) for c in candidates]
scores = reranker.predict(pairs)             # expensive, accurate
top = [c for _, c in sorted(zip(scores, candidates), key=lambda x: -x[0])[:5]]

Two current gotchas. The canonical model id is now cross-encoder/ms-marco-MiniLM-L6-v2 — note there is no dash before the 6, unlike the L-6 form you will find in older posts. And sentence-transformers 5.4 renamed max_length to max_seq_length and tokenizer_kwargs to processor_kwargs, and made constructor arguments keyword-only; copied snippets from 2024 will throw.

Embeddings

If you are still on all-MiniLM-L6-v2 because a tutorial used it, that is worth revisiting — 384 dimensions and an effective context around 256 tokens is limiting for anything but short passages. google/embeddinggemma-300m is the closest drop-in upgrade; Qwen/Qwen3-Embedding-0.6B is stronger if you can afford it.

Resist the urge to pick by leaderboard. MTEB is widely regarded as partially contaminated, and leaderboard rank correlates poorly with performance on your specific corpus. You already built a ground-truth set — that is your benchmark. Swap the model, re-run recall@k, keep the winner.

Adding context to chunks

The technique with the best-documented results here is contextual retrieval: before embedding each chunk, prepend a short generated description of where that chunk sits in its parent document. The published benchmark (Anthropic, September 2024) reports retrieval failure rate — defined as 1 minus recall@20 — dropping from a 5.7% baseline to 3.7% with contextual embeddings (a 35% reduction), 2.9% adding contextual BM25 (49%), and 1.9% adding reranking (67%).

Those are real, verifiable figures, and the technique is genuinely effective. But the caveats are stripped from nearly every citation of them, so: that headline is the average over five datasets for a single embedding model (the best performer of the six tested), not across all models. Per-dataset variance is large and inconsistent in direction — on one dataset, plain embeddings + BM25 beat contextual embeddings outright. Reranking is not monotonically good either; at recall@10 on one set it hurt for four of six models. And the 67% is a recall@20 result that does not transfer to smaller k. It is a vendor engineering blog: no error bars, no significance tests, eval sets unreleased. Treat it as a strong prior for an experiment on your corpus, not as a guaranteed 67%.

The cheap version costs nothing and often captures much of the gain: prepend the document title and section breadcrumb to each chunk before embedding.

# Cheap contextualization — no LLM call per chunk
enriched = f"{doc_title} > {section_path}\n\n{chunk_text}"

Things everyone repeats that don’t survive checking

This is the section I most want you to read, because acting on a laundered statistic is how you end up fixing the wrong thing.

”70–73% of RAG failures are retrieval failures”

This is everywhere. It is also, as far as I can determine, fabricated.

I traced it through four hops. A widely-cited page asserts “73% of RAG failures originate at the retrieval stage” and cites a Medium post — where 73% appears only in the title and never in the body — which links to a third blog whose headline says 73% while its own body says 72%, sourced to “according to recent internal research.” No link, no institution, no methodology. That is the origin: an uncited number that could not even keep itself consistent between its headline and its body.

Two separate things are wrong here. The root number is uncited, and the claim mutates in transit: the original assertion is about how many RAG deployments fail in production — an entirely different proposition from what share of failures are retrieval-attributable. Somewhere in the chain, one became the other.

It is often attributed to Barnett et al., “Seven Failure Points When Engineering a RAG System”. That paper defines seven failure points and reports no percentage breakdown by category at all. During research, a search engine’s own AI summary confidently told me the 73% figure “comes from research by Barnett et al.” — an invented citation. The laundering has reached the retrieval layer. Don’t trust a citation you haven’t opened.

The closest real number I found comes from a doctoral thesis synthesising 150+ RAG deployment papers (arXiv:2605.23024), which attributes 40.9% of failures to retrieval, 28.2% to augmentation, and 30.9% to generation. Retrieval is the largest single category — a plurality, not a supermajority. And that is a literature synthesis, not a measurement of your system.

Which is the actual point: the distribution for your system is whatever your oracle test says it is. That is why the test comes first. You do not need a population statistic; you need ten probes and an afternoon.

”Chunk overlap adds no measurable benefit”

This one is real but routinely over-claimed. The source is a genuine peer-reviewed ECIR 2026 paper (arXiv:2601.14123): “Overlap adds cost without measurable gains. Across paired configurations, adding 10–20% overlap did not improve BERTScore or EM (e.g., |ΔBERTScore| ≤ 0.004; EM differences ≤ 0.001).”

Before you rip overlap out of your pipeline, read the scope. It is a three-page short paper testing one dataset (Natural Questions), one retriever (SPLADE — sparse), and two overlap settings (0% and 20%, not a sweep). Dense embedding retrievers, which most production RAG actually uses, were never tested. The paper concedes its results “are not representative of general text-centric corpora.” And its explanation of the null result is itself conditional: with a sentence-aware pipeline and a sparse retriever, boundary spillover rarely changes the retrieved content — which does not obviously transfer to naive fixed-size chunking, exactly where overlap is conventionally used.

The authors’ own recommendation is the honest version: use zero overlap unless you have evidence your retriever benefits from boundary redundancy. Overlap is not free — it inflates your index and duplicates content across chunks — so it is a reasonable default to drop. Just measure it rather than citing the paper as a universal law.

”Semantic chunking gets 91.9% recall but only 54% accuracy, vs 69% for recursive”

Do not repeat this sentence. Both numbers are individually real, and no single study produced them together.

The 91.9% comes from a Chroma technical report on chunking evaluation, which measures token-level recall — and contains no end-to-end accuracy metric whatsoever. (Its own figure is 91.9% ± 26.5%; the variance is always dropped.) The 54%/69% comes from an unrelated vendor blog with n = 30 synthetic Q&A pairs. A 69-vs-54 gap on 30 questions is about four questions — statistically indistinguishable from noise. And that post self-invalidates: it attributes semantic chunking’s poor showing to a poorly tuned cosine threshold. So the 54% measures a misconfigured chunker, not semantic chunking.

Welding them with a “but” invents a causal story — “retrieves well, answers badly” — that neither study measured. The real, defensible finding on semantic chunking is duller: a peer-reviewed paper asking Is Semantic Chunking Worth the Computational Cost? concluded that “the computational costs associated with semantic chunking are not justified by consistent performance gains.” That is your prior. Start with recursive chunking.

”Just increase top-k”

Recall@k rises monotonically with k. Answer accuracy does not. These are different curves and conflating them is why bumping k sometimes makes things worse.

NVIDIA’s OP-RAG paper (arXiv:2409.01666) documents the inverted U directly: “as the number of retrieved chunks increases, the answer quality initially rises, and then declines.” Llama3.1-70B on ∞Bench EN.QA peaked at 47.25 F1, then declined to 44.43 as chunks kept coming.

The mechanism is exactly what you would predict from the distractor research: raising k pulls in high-similarity near-misses — chunks that score well on cosine distance precisely because they are topically adjacent, and that are therefore the most confusing possible distractors. Wu et al. (COLM 2024) found LLMs are distracted most by semantically related irrelevant content that scores highly on similarity metrics. You are not adding coverage; past the peak, you are adding traps.

Find your own peak. Sweep k over [3, 5, 10, 20, 50] and measure end-to-end accuracy — not recall — at each.

One more correction while we are here: “Lost in the Middle” is frequently cited as evidence that higher k degrades accuracy. It does not show that. It shows saturation — “using 50 documents instead of 20 retrieved documents only marginally improves performance (∼1.5% for GPT-3.5-Turbo).” More k still helped; it just stopped being worth the tokens.

”Random noise improves RAG accuracy”

The famous “Power of Noise” result — that adding random documents boosted accuracy substantially — failed to replicate. A July 2026 reproduction (arXiv:2607.03615) concludes “the original effect cannot be robustly confirmed as a general benefit of noisy retrieval under these experimental conditions,” tracing much of it to truncation artifacts and malformed generations under constrained decoding. Do not add random documents to your context. The distractor half of that paper, cited earlier, held up fine.

Tooling traps that will cost you an afternoon

These are current as of July 2026 and each one bites on first contact.

Ragas does not install cleanly. pip install ragas gives you 0.4.3, which declares an unpinned langchain-community — and recent langchain-community removed a module it imports. A fresh install fails at import with ModuleNotFoundError: No module named 'langchain_community.chat_models.vertexai'. Working pin set:

pip install "ragas==0.4.3" "langchain-community==0.3.31" "langchain-core<0.4" \
            "langchain<0.4" "langchain-openai<0.4" "langchain-text-splitters<0.4"

This pins you to the LangChain 0.3 line, which cannot coexist with LangChain 1.x. Use a separate virtualenv for evaluation, not your app env.

Ragas’s own deprecation warning points somewhere that doesn’t work. It tells you to migrate to ragas.metrics.collections. But collections metrics are not Metric subclasses, and evaluate() rejects them with TypeError: All metrics must be initialised metric objects. So: use the legacy ragas.metrics imports with evaluate(), and collections only for standalone await metric.ascore(...). Note the class names have drifted from the metric names people quote — answer_relevancy is ResponseRelevancy, context_recall is LLMContextRecall:

from ragas import evaluate, EvaluationDataset
from ragas.metrics import Faithfulness, LLMContextRecall, ContextPrecision, ResponseRelevancy

ds = EvaluationDataset.from_list([
    {
        "user_input": "What is the refund window?",
        "retrieved_contexts": ["Refunds are accepted within 30 days of purchase."],
        "response": "You can request a refund within 30 days.",
        "reference": "30 days",
    },
])

result = evaluate(ds, metrics=[Faithfulness(), LLMContextRecall(),
                               ContextPrecision(), ResponseRelevancy()], llm=your_llm)

from langchain.text_splitter import ... is removed, not deprecated. Countless sources call it deprecated; it is gone — a hard ModuleNotFoundError. Use the standalone langchain-text-splitters package, which does not depend on langchain at all:

from langchain_text_splitters import RecursiveCharacterTextSplitter

Relatedly, langchain-experimental was sunset in May 2026, and SemanticChunker’s migration to a supported package was closed as not planned — it is a dead end. If you want semantic chunking, look at chonkie or LlamaIndex’s SemanticSplitterNodeParser.

pgvector’s metadata filter silently returns too few rows. This is the nastiest one, because it does not error. With HNSW and the default ef_search = 40, a filter matching 10% of your rows and a LIMIT 10 returns roughly 4 rows — no warning. At 1% selectivity you often get zero. The index walks the graph and then applies your filter, so most candidates are discarded after the fact.

Raising ef_search is a bandaid (it caps at 1000). The real fix is iterative scan, available since pgvector 0.8.0 and off by default:

SET LOCAL hnsw.iterative_scan = relaxed_order;
SET LOCAL hnsw.ef_search = 100;

Use SET LOCAL, not plain SET — a plain SET leaks across connections when you are behind PgBouncer, which produces a genuinely baffling class of intermittent bug.

While you are there: pgvector is at 0.8.5 as of July 2026, and upgrading matters beyond features. 0.8.2 fixed a buffer overflow (CVE-2026-3172) in parallel HNSW builds, and 0.8.3/0.8.4 fixed HNSW index corruption during VACUUM. If you are on 0.8.1 or earlier in production, that is a real problem wearing a version number.

A workflow that converges

Putting it together, the loop that actually terminates:

1. Collect 30-50 REAL failing queries from logs.        (not synthetic)
2. Run the oracle test on 10 of them.                   (10 minutes)
3. Read the verdict distribution.
      ├─ mostly GENERATION  → prompt / model / k reduction
      ├─ mostly DILUTION    → rerank, lower k, filter
      └─ mostly RETRIEVAL   → step 4
4. Check the top-50 pool for each miss.
      ├─ chunk present  → RANKING     → add cross-encoder rerank
      └─ chunk absent   → REACHABILITY → hybrid search, then chunking,
                                          then embeddings
5. Change ONE thing. Re-measure recall@k on the same set.
6. Sweep k for end-to-end accuracy. Find your peak. It is not 20.
7. Repeat from 2. The distribution will have moved.

The discipline that makes this work is step 5. One change, same ground-truth set, re-measure. The reason RAG debugging feels like guesswork is that people change three things at once and then cannot attribute the result — and because a re-index between each attempt makes the loop so slow that patience runs out before evidence arrives.

The reason to run the oracle test first is that it is the cheapest possible way to eliminate half the search space. Ten minutes of work tells you whether an entire category of fixes — everything about chunking, embeddings, and vector databases — is even capable of helping you.

Common mistakes

• Fixing before diagnosing. The default failure mode, and the expensive one.
• Synthetic ground-truth queries. They share vocabulary with the source chunk and make retrieval look better than it is.
• Reporting the mean. It hides an entirely broken query class behind a decent average.
• Adding a reranker when the chunk isn’t in the pool. A reranker reorders; it cannot conjure.
• Changing several things at once, then being unable to attribute the improvement — or the regression.
• Trusting a statistic you didn’t open. Including any in this article. The arXiv IDs are all linked; go read them.
• Evaluating on the queries you already fixed. Hold out a set, or you are measuring your own memory.

Expert tips

• Keep the oracle test in CI as a handful of probes. It catches “someone changed the prompt template” the day it happens rather than the week after.
• Log the retrieved chunk IDs and scores with every production answer. Debugging RAG without retrieval logs is debugging with the lights off, and you cannot reconstruct them after the fact.
• When a user reports a bad answer, your first action is not to read the answer. It is to read what was retrieved.
• Test with the ugliest real queries you have — typos, acronyms, one-word questions. That is where hybrid search earns its keep and where pure dense retrieval quietly collapses.
• Prefer a small, honest ground-truth set you actually maintain over a large synthetic one you trust more than it deserves.

Where this is heading

Two things are worth watching. Interactive debugging tooling is finally getting research attention — the re-indexing bottleneck that RAGGY identified is the single biggest reason people abandon systematic debugging for guessing, and that is a solvable engineering problem.

The more interesting shift is that long-context models have not made retrieval obsolete, and the evidence increasingly says the opposite. The OP-RAG result is pointed here: a 70B model doing RAG over 48K tokens of retrieved context beat the same model reading 117K tokens of full context, 47.25 F1 to 34.26. Feeding everything to a big context window is not just more expensive — it scored worse. Retrieval quality is not a transitional concern that scale will retire. It is the thing that determines whether the model gets a clean signal or a haystack.

Which brings this back to where it started. The reason to diagnose before you fix is not methodological tidiness. It is that RAG has five stages that all fail silently and produce identical-looking output, and the only way to tell them apart is to run the experiment that separates them. Ten minutes, one script, and you know which half of the internet’s advice to ignore.

FAQ

Is my RAG problem retrieval or generation? Run the oracle test. Take a question your system gets wrong, find the chunk that actually contains the answer, and hand that chunk to the model directly — bypassing retrieval entirely. If the answer is now correct, retrieval is your bug. If it’s still wrong with the answer sitting in the context window, your prompt or generation step is the bug. This takes about ten minutes and it is the single highest-value thing you can do, because the two failures have completely disjoint fix lists. Chunking, embeddings, and reranking cannot fix a generation bug, and no amount of prompt tuning fixes a chunk that was never retrieved.

What is a good recall@k score for a RAG system? There is no universal threshold, and anyone who gives you one without naming a dataset is guessing. Recall@k depends on your corpus, your queries, and how you define a relevant chunk. What matters is the shape of your own curve. Measure recall at k=1, 5, 10, and 20 on your own ground-truth set. If recall@20 is high but recall@5 is much lower, your retriever finds the right chunk but ranks it poorly — add a reranker. If recall@20 is also low, the chunk is not reachable at all, and reranking cannot help you because a reranker only reorders what retrieval already returned.

Why does my RAG hallucinate even when the right context is retrieved? Two common causes. First, distractors: the correct chunk was retrieved alongside several near-miss chunks that look semantically similar, and the model latched onto the wrong one. Cuconasu et al. (SIGIR 2024) found that adding even a single distracting document caused accuracy drops of up to 25 percent. Second, prior conflict: when retrieved context contradicts what the model already believes, it does not always defer to the context. The ClashEval study (arXiv:2404.10198) measured how often models adopt incorrect retrieved content over their own correct knowledge, and the rate ranged from about 31 percent to 63 percent depending on the model.

Why did increasing top-k make my RAG worse? Because retriever recall and answer accuracy are different curves. Recall@k rises monotonically with k — more chunks can only mean more chances to include the right one. Answer accuracy does not. NVIDIA’s OP-RAG paper (arXiv:2409.01666) documented an inverted-U: quality rises with more retrieved chunks, peaks, then declines. Raising k pulls in exactly the chunks that are most dangerous — high-similarity near-misses that score well on cosine distance but do not answer the question. You are trading a retrieval problem for a distractor problem.

Does chunk overlap improve RAG retrieval? Less than the conventional advice assumes, and possibly not at all in some setups. An ECIR 2026 paper (arXiv:2601.14123) found that adding 10–20 percent overlap produced no measurable gain, with BERTScore differences at or below 0.004. But read the scope before you rip overlap out: that result used a sentence-aware chunker and a sparse (SPLADE) retriever on Natural Questions. Dense embedding retrievers — what most production RAG actually uses — were not tested. The authors’ own recommendation is conditional: use zero overlap unless you have evidence your retriever benefits from boundary redundancy. Measure it on your corpus.

How many queries do I need in a RAG ground-truth set? Start with 30 to 50 real queries and expect to grow. The number matters less than the sourcing: pull them from actual user logs and failure reports, not from your imagination or from an LLM asked to invent questions about your docs. Synthetic questions tend to be phrased using the same vocabulary as the source chunk, which makes retrieval look far better than it is — the embedding match is nearly lexical. Real users ask questions with different words than your documents use, and that vocabulary gap is precisely what you need to measure.