Week 10 · Phase 2 Engineering & Tools
The Evaluation Lab
“It looks right” is not a metric. Week 10 turns your RAG pipeline into something you can score, compare, and defend — faithfulness, relevancy, precision, recall.
The four questions every RAG eval asks
Each RAGAS metric interrogates a different link in the chain — two judge the retriever, two judge the generator.
Context Precision
“Was what we fetched actually relevant?”Of the retrieved chunks, how many were useful? Low = your prompt is full of noise. Judges the retriever.
Context Recall
“Did we fetch everything we needed?”Of all relevant chunks in the corpus, how many did we find? Low = answers miss facts. Judges the retriever.
Faithfulness
“Is every claim grounded in the context?”Claims in the answer supported by retrieved text ÷ total claims. Low = hallucination. Judges the generator.
Answer Relevancy
“Did we answer the question that was asked?”A faithful, precise answer to the wrong question still fails the user. Judges the generator.
Precision@k vs recall — the eternal trade-off
Our corpus has 10 documents; 4 are truly relevant (leaf icon) to the query. The retriever returns documents in its own ranked order. Drag k and watch the two metrics fight.
Your Week 8 acceptance bar was precision ≥ 0.7 — now you can see what hitting it costs in recall. Dashed red tiles are relevant documents your retriever missed at this k.
The faithfulness check, by hand
This is literally what RAGAS automates: split the answer into claims, then hunt for each one in the retrieved context. Run it.
Reading the scores: failure modes → fixes
Week 10 deliverable: an evaluation report with an improvement plan — score your pipeline, name the weakest metric, and prescribe the fix from this table.