Weekend Challenge · Week 5
The Failure Audit
Play quality engineer for your own system: every bad answer gets a root cause, and every root cause gets a ticket.
Your mission
- Run your full golden dataset through the Week 4 pipeline
- Label every failure: retrieval, generation, or chunking
- Write the top 3 'tickets': the highest-impact fixes, with evidence
Deliverable
failure_audit.md: counts per category, example transcripts, and your 3 tickets, committed.
Stretch goal
Have an LLM judge label the same failures independently. Compute agreement with your human labels. Where does the judge get it wrong — and would you still trust it to gate deploys?