Weekend Challenge · Week 1
Catch Your Model Lying
Before trusting retrieval to fix hallucination, you need to see hallucination — systematically, in your own model, with your own eyes.
Your mission
- Write
hallucination_probe.py: ask your local model 20 questions — 5 each across four categories: stable general knowledge, niche long-tail facts, post-training-cutoff events, and your capstone domain - Score every answer yourself: ✅ correct, ❌ wrong, 🤷 refused/hedged
- Write a 5-sentence conclusion: in which categories does retrieval earn its keep?
Deliverable
A results table (markdown or CSV) + conclusion, committed with message Week 1 challenge: hallucination probe.
Stretch goal
Re-run all 20 questions at temperature 0 and 1.0. Does any category change verdict? What does that imply about 'just lower the temperature' as a hallucination fix?