zorost/ai-engineering-lab/rag-pipeline-checkup

RAG Pipeline Checkup

1 · Purpose

Debug a RAG system as five separable stages, so "the answer was wrong" becomes "retrieval missed the policy section" or "the model ignored the citation", each with its own fix.

2 · When to use

• Building any RAG feature, or when a RAG feature's answers are wrong, ungrounded,

or stale.

• Before adding complexity (HyDE, hybrid search, graphs) to a pipeline whose basics

are unmeasured.

3 · Inputs

• The document corpus and the chunking code/config.
• A question set: ≥ 20 real questions with the passage that should answer each

(the retrieval golden set, build it with eval-first-development).

• The ability to log intermediate stage outputs.

4 · Procedure

1. Chunking. Print 10 random chunks. Check: each chunk is one topic, carries its

source and section, and respects the size budget (~300 to 500 tokens with 10 to 15% overlap is the default start). Fix splits that cut tables or definitions in half.

1. Embedding. Embed three paraphrases of one question. Confirm they retrieve

each other's chunks, if paraphrases diverge wildly, the embedding model does not match the domain. Try a domain-appropriate model before tuning anything else.

1. Retrieval. Score the golden set: for each question, is the right passage in

the top-k? Record recall@5. Below 0.8, fix retrieval before touching generation, a model cannot cite what it never saw.

1. Reranking. Add a reranker over the top-20 and re-measure recall@5 of the

reranked top-5. Keep it only if recall rises; record the latency cost either way.

1. Generation. Require citations: every factual sentence must reference a

retrieved chunk ID, and the answer must say "not in the documents" when retrieval came back empty. A schema constrains shape, never truth, spot-check that cited chunks actually contain the claim.

1. End-to-end eval. Score the full pipeline on the question set with the answer

grader. Record: retrieval recall, answer score, citation precision (sampled).

1. Attribute every failure to exactly one stage. Fix the stage with the most

failures first. Re-run the full eval after each fix.

5 · Anti-rationalization

6 · Red flags

• No recall number exists for retrieval alone.
• Chunks lack source metadata, so citations are impossible by construction.
• The answer never says "not in the documents", the model always finds something.
• A fix was applied to generation while retrieval recall sat below 0.5.

7 · Verify

• recall@5 on the retrieval golden set is recorded, before and after each change.
• The end-to-end answer score is recorded against the same question set.
• A sample of 10 answers has citation precision checked by hand (cited chunk

actually supports the claim).

• Every known failure is attributed to one stage in the log.

8 · ZoroLogistics example

Week 7's policy bot: 20 ops questions ("who approves a refund over $500?") with the handbook section that answers each. Baseline: recall@5 = 0.65, the chunker split the approval table across two chunks. Fix: table-aware chunking → recall 0.90, answer score 0.61 → 0.88 without touching the model. The lesson the pipeline teaches: retrieval was the ceiling all along.

© 2026 Zorost Intelligence LLC · zorost.com