Evaluation: Diagnosing Failures Systematically

Until now you've measured retrieval. This week you measure answers — and build the machine that tells you why a bad answer was bad. This is the week that separates people who tinker with RAG from people who ship it: without diagnosis, every fix is a guess, and guesses that happen to work are indistinguishable from guesses that don't.

Part 1 — The failure taxonomy: three root causes, three different fixes

A user reports: "the answer was wrong." That report is useless until you localize it. Every wrong RAG answer has exactly one of three root causes, and the fixes share nothing:

The diagnostic order is fixed and non-negotiable: check retrieval first, chunking second, generation last. Most teams debug backwards — they rewrite prompts for hours to fix what was a retrieval miss. The prompt was never the problem.

Part 2 — Metrics for the generation half

Recall@k and MRR say nothing about the text the user actually reads. Four metrics complete the picture — note how each isolates a different stage:

The pairing to internalize: context recall bounds what's possible; faithfulness measures whether the model honored it. High recall + low faithfulness = a prompting problem. Low recall + high faithfulness = a model faithfully answering from the wrong evidence — the most dangerous quadrant, because the answer will be confidently, citably wrong.

Part 3 — LLM-as-judge, and its biases

Human grading doesn't scale past a few dozen answers. The standard workaround is to have a strong model grade the output — and it works, provided you know what the judge is bad at:

• Verbosity bias: longer answers score higher, whether or not they're better.
• Self-preference: judges favor text in their own style — a real problem when the judge and generator are the same model.
• Position bias: in A/B comparisons, whichever is shown first tends to win. Always test both orders.
• Poor calibration: ask for 1–10 and you'll get 7s and 8s forever. Binary or 3-point scales are far more reliable.

The practical rules: judge one narrow question at a time (never "rate this answer overall"), decompose the answer into claims and check each against the context, use binary verdicts, and — the discipline everyone skips — calibrate against 20 hand-labeled examples. If the judge agrees with you less than ~80% of the time on your own domain, its scores are decoration.

Part 4 — The golden dataset

Fifty carefully labeled questions beat five thousand noisy ones. Build yours with a synthesize-then-verify loop: have the LLM draft candidate questions from each document (cheap, gets coverage), then you verify and fix every one (slow, gets truth). Never ship a golden item you haven't read.

Deliberately include the hard cases — the ones that catch regressions no happy-path set will:

• Unanswerable questions your corpus genuinely cannot answer. The correct behavior is refusal; measuring it is how you catch a system that never says "I don't know."
• Multi-source questions needing two or more documents (they'll matter enormously next week).
• Near-miss questions whose vocabulary matches a wrong document — the trap your retriever is most likely to fall into.

Lab — build the measurement machine

Step 1 · A generation step to evaluate

You need answers before you can grade them. Create answer.py — a deliberately minimal RAG generator (Week 6 does this properly):

import ollama
from pathlib import Path
from rerank import reranked_search

TEXTS = {p.name: p.read_text(errors="ignore")
         for p in Path("corpus").glob("*") if p.suffix in (".txt", ".md")}

PROMPT = """Answer the question using ONLY the context below.
If the context does not contain the answer, say exactly: I don't know.

Context:
{context}

Question: {question}
Answer:"""

def answer(question, k=3):
    docs = reranked_search(question, k=k)
    context = "\n\n---\n\n".join(f"[{d}]\n{TEXTS[d][:2000]}" for d in docs)
    r = ollama.chat(model="llama3.1:8b", messages=[
        {"role": "user", "content": PROMPT.format(context=context, question=question)}])
    return r["message"]["content"].strip(), docs, context

Step 2 · Golden dataset, synthesized then verified

Create make_golden.py to draft candidates:

import json, ollama
from pathlib import Path

items = []
for p in list(Path("corpus").glob("*"))[:25]:
    if p.suffix not in (".txt", ".md"): continue
    r = ollama.chat(model="llama3.1:8b", messages=[{"role": "user", "content":
        "Read the document and write 2 specific questions a real user would ask that "
        "THIS document answers. One question per line, no numbering.\n\n"
        + p.read_text(errors="ignore")[:3000]}])
    for q in r["message"]["content"].strip().splitlines():
        q = q.strip("-• ").strip()
        if len(q) > 10:
            items.append({"question": q, "source": p.name, "ground_truth": "", "verified": False})

json.dump(items, open("golden_draft.json", "w"), indent=2, ensure_ascii=False)
print(f"{len(items)} candidates drafted — now verify them by hand")

Then do the unglamorous part: open golden_draft.json, keep ~40 good items, write the correct ground_truth for each, fix wrong source labels, delete nonsense, and hand-add 5 unanswerable + 3 multi-source + 3 near-miss questions. Save as golden.json with verified: true. This file is the most valuable artifact you build this week — it gates every change you make for the rest of the course.

Step 3 · The judge

Create judge.py — claim-level faithfulness plus relevance, both binary:

import ollama

def ask(prompt):
    r = ollama.chat(model="llama3.1:8b", messages=[{"role": "user", "content": prompt}])
    return r["message"]["content"].strip().upper()

def faithfulness(answer_text, context):
    claims = [c.strip() for c in answer_text.split(".") if len(c.strip()) > 15]
    if not claims: return 1.0
    supported = 0
    for c in claims:
        v = ask(f"Context:\n{context[:4000]}\n\nClaim: {c}\n\n"
                f"Is this claim fully supported by the context? Reply YES or NO only.")
        supported += v.startswith("YES")
    return supported / len(claims)

def relevance(question, answer_text):
    v = ask(f"Question: {question}\nAnswer: {answer_text}\n\n"
            f"Does the answer address the question asked? Reply YES or NO only.")
    return 1.0 if v.startswith("YES") else 0.0

Step 4 · Calibrate the judge before trusting it

Non-negotiable. Run 20 golden items, record the judge's verdicts, then grade the same 20 yourself and compute agreement:

agreement = matching_verdicts / 20
# ≥0.8  → usable for bulk evaluation
# <0.8  → tighten the prompt (shorter claims, stricter wording) and re-calibrate

Write the agreement number in your README next to every judge-derived metric. A metric without its calibration is a rumor.

Step 5 · The failure audit

Create audit.py — run the full golden set and auto-classify every failure by the Part 1 taxonomy:

import json
from answer import answer
from judge import faithfulness, relevance

golden = json.load(open("golden.json"))
rows = []
for item in golden:
    ans, docs, ctx = answer(item["question"])
    retrieved_ok = item["source"] in docs                 # was the right doc retrieved?
    f = faithfulness(ans, ctx)
    r = relevance(item["question"], ans)
    if not retrieved_ok:            cause = "retrieval"
    elif f < 0.7:                   cause = "generation"
    elif r == 0:                    cause = "generation"
    else:                           cause = "ok"
    rows.append({**item, "answer": ans, "docs": docs,
                 "faithfulness": f, "relevance": r, "cause": cause})

json.dump(rows, open("audit_results.json", "w"), indent=2, ensure_ascii=False)
from collections import Counter
print(Counter(r["cause"] for r in rows))

The script separates retrieval from generation automatically. Chunking failures it cannot see — those need your eyes: pull 5 cases labeled "generation", read the retrieved chunk cold, and ask whether a careful human could have answered from it. If not, relabel: chunking. That's the manual step no framework does for you.

Step 6 · Write the three tickets

Turn the audit into failure_audit.md: counts per cause, two example transcripts each, and the three highest-impact fixes with evidence. Not "improve retrieval" — "9 of 14 retrieval failures were paraphrase queries; add query rewriting (Week 4, measured +0.2 R@5 on 3 test queries)." That is what an engineering ticket looks like.