Why Your RAG Pipeline Is Lying to You

Here's an uncomfortable truth: your RAG pipeline is probably retrieving the wrong information a third of the time.

Not wrong enough to crash. Not wrong enough to obviously fail. Just wrong enough to produce confident-sounding responses that are subtly, dangerously incorrect.

I've audited dozens of RAG implementations. The pattern is consistent: teams build pipelines, test them on a few happy-path examples, declare success, and ship. Six months later, users are complaining that the system "doesn't really work" and nobody knows why.

The problem isn't the LLM. It's retrieval. And almost nobody is measuring it.

Let me show you what I mean.

The LLM saw one relevant chunk out of five. It improvised with its parametric knowledge to fill the gaps. The answer looks right but contains details that aren't in your knowledge base.

This happens 30-40% of the time in typical RAG deployments.

Three reasons this failure mode persists:

Here's the evaluation framework I use:

Step 1: Build a Retrieval Test Set

Create a dataset of queries paired with the correct source documents.

retrieval_test_set = [
    {
        "query": "How do I reset my password?",
        "relevant_doc_ids": ["doc_123", "doc_456"],
        "relevant_chunks": ["chunk_123_2", "chunk_456_1"]
    },
    {
        "query": "What's the refund policy for annual subscriptions?",
        "relevant_doc_ids": ["doc_789"],
        "relevant_chunks": ["chunk_789_3", "chunk_789_4"]
    },
    # 50-100 examples covering your key use cases
]

Step 2: Measure Retrieval Quality

For each test query, run your retrieval and measure:

def evaluate_retrieval(test_set, retriever, k=5):
    results = []

    for test in test_set:
        # Run retrieval
        retrieved = retriever.retrieve(test["query"], top_k=k)
        retrieved_ids = [chunk.id for chunk in retrieved]

        # Calculate metrics
        relevant_set = set(test["relevant_chunks"])
        retrieved_set = set(retrieved_ids)

        # Precision: What fraction of retrieved chunks are relevant?
        precision = len(relevant_set & retrieved_set) / len(retrieved_set)

        # Recall: What fraction of relevant chunks did we retrieve?
        recall = len(relevant_set & retrieved_set) / len(relevant_set)

        # Hit rate: Did we get at least one relevant chunk?
        hit = len(relevant_set & retrieved_set) > 0

        # MRR: Where does the first relevant chunk appear?
        mrr = 0
        for i, chunk_id in enumerate(retrieved_ids):
            if chunk_id in relevant_set:
                mrr = 1 / (i + 1)
                break

        results.append({
            "query": test["query"],
            "precision": precision,
            "recall": recall,
            "hit": hit,
            "mrr": mrr
        })

    return results

If your hit rate is below 90%, you have a serious problem. If precision is below 50%, you're flooding the LLM with noise.

Step 3: Analyze Failure Patterns

Don't just measure—understand why retrieval fails.

When retrieval quality drops, run through this:

If you do nothing else:

That's a few hours of work. It will save you months of debugging mysterious answer quality issues.