AXLE · RAG & Context Engineering
Home / Syllabus / Week 5
Week 5 of 8

Evaluation: Diagnosing Failures Systematically

'It looks right' is not evaluation. This week you build the measurement machine that turns debugging from guesswork into diagnosis.

Learning objectives
  • Distinguish retrieval failures from generation failures from chunking failures
  • Implement faithfulness, answer relevance, and context precision/recall metrics
  • Use LLM-as-judge responsibly, knowing its biases
  • Build a golden dataset cheaply and maintain it
Study material

Open the full Week 5 study material → Complete lesson: concepts in depth, the full lab with all code, and troubleshooting.

Concepts

1. The failure taxonomy

A wrong answer has exactly three root causes: the right context never arrived (retrieval failure), it arrived but the model ignored or contradicted it (generation failure), or it arrived mangled (chunking failure). Each has a different fix — diagnosis must come first.

2. Generation metrics

Faithfulness: is every claim in the answer supported by the retrieved context? Answer relevance: does it address the question? Context precision/recall: was the retrieved context the right context? Together these localize the failure.

3. LLM-as-judge

Using a strong model to grade answers scales evaluation dramatically — but judges prefer verbose answers, their own phrasing, and the first option shown. You'll learn the calibration rituals that keep judge scores honest.

4. Golden datasets

Fifty carefully labeled question-answer-source triples beat five thousand noisy ones. You'll build yours with a synthesis-then-verify loop over your own corpus.

Lab — live session

  1. Build an evaluation harness (RAGAS or hand-rolled) over your Week 4 pipeline
  2. Generate a golden dataset from your corpus; hand-verify every item
  3. Run a full failure audit: label every bad answer with its root cause
  4. Set up the harness to run on every future pipeline change
Checkpoint

An eval suite plus a failure taxonomy for your system. This is your debugging compass for the rest of the program.

Reading

RAG Evaluation Metrics — Confident AIAnswer relevancy, faithfulness, contextual precision and recall, clearly explained.RAGAS documentation — FaithfulnessHow the leading open-source framework computes it.

Watch

RAG Evaluation Metrics Explained: Context Precision, Recall, Relevancy & FaithfulnessEach core metric with worked examples.

Reflection — bring answers to the next session

  1. Of your failures, what fraction were retrieval vs. generation vs. chunking? Did that surprise you?
  2. Where would an LLM judge disagree with your human judgment on your own domain?
  3. What is the smallest golden dataset you'd trust to gate a production deploy?
Weekend challenge

Challenge 5: The Failure Audit — Play quality engineer for your own system: every bad answer gets a root cause, and every root cause gets a ticket.

← Week 4Week 6 →