When (and When Not) to Retrieve

This is the complete Week 1 lesson. In the live session we work through it together; between sessions it's your reference. By the end you'll have a professional environment running a local LLM — and a measured map of exactly where that model can't be trusted.

Part 1 — How a language model "knows" things

During training, a model reads a vast snapshot of text and compresses statistical patterns from it into billions of numeric weights. Two properties of that process explain almost everything this course exists to fix:

The compression is lossy. Facts that appear thousands of times in training data (the capital of France, the boiling point of water) survive compression with high fidelity. Facts that appear rarely — the prerequisites of a specific AULA course, clause 7 of your supplier contract, anything published last month — blur or vanish. The model didn't choose what to forget; frequency chose for it.

There is no "I don't know" flag. The model's only operation is: given the text so far, produce a plausible continuation. When the underlying knowledge is missing, the mechanism doesn't change — it still produces a plausible continuation. That's a hallucination: not a malfunction, but the normal mechanism running without data. This is why hallucinations are fluent, confident, and formatted exactly like correct answers.

ollama run llama3.1:8b "What is the capital of Australia?"
ollama run llama3.1:8b "What were the exact enrollment requirements of AULA's 2025 astrobiology program?"

Part 2 — The context window: knowledge that arrives at runtime

Everything the model didn't memorize must arrive through the prompt — the context window. Think of it as the model's working memory: whatever text you place there, the model can attend to when generating. This is the escape hatch from the limits of Part 1: the model doesn't need to know your document if you show it your document.

But the window is a budget, measured in tokens (a token ≈ ¾ of an English word). Three budget rules govern everything we build in this course:

1. It's finite. Your corpus won't fit. Something must choose which fragments enter — that chooser is the retriever, and its quality bounds the whole system.
2. Quality degrades as it fills. Models attend unevenly across long contexts; stuffing the window hurts both cost and accuracy. More context is not more better.
3. Every token costs. Latency and (with hosted models) money scale with context length. Production systems are engineered around this line item.

Context engineering — the second noun in this course's title — is the discipline of spending that budget deliberately: what gets in, in what order, in what format, and what stays out.

Part 3 — The retrieval decision

Retrieval-augmented generation adds a step before generation: search an external knowledge store, place the best findings in the context window, then generate. It is an engineering trade — you buy grounding and pay in latency, complexity, and new failure modes. The professional skill is knowing when the trade is worth it.

Real systems receive all six kinds of traffic. That's why mature architectures route: a cheap decision layer classifies each query and only invokes retrieval when it pays. You'll build exactly this intuition in the lab — and by Week 7, your agent will make this decision by itself, per query.

Part 4 — The alternatives, honestly compared

RAG is not the only way to close a knowledge gap. You should be able to argue all four options — this table is a favorite architecture-review and interview question:

Note the pattern: these compose rather than compete. Production systems routinely fine-tune for format, retrieve for facts, and call tools for structured lookups — in the same request.

Lab — your professional setup

From day one you work like a working engineer: local model, reproducible environment, version control. Every command below runs in your Mac's Terminal.

Step 1 · Ollama and models

brew install ollama          # or download the app from ollama.com
ollama pull llama3.1:8b      # generator (~5 GB, needs ~8 GB RAM)
ollama pull nomic-embed-text # embedder — we'll need it from Week 3

Verify: ollama run llama3.1:8b "Say hello in one sentence."

Step 2 · Python environment with uv

Professionals never install into system Python — environments are isolated and reproducible, so "works on my machine" means it works on every machine.

brew install uv
mkdir -p ~/Desktop/AXLE/rag-course && cd ~/Desktop/AXLE/rag-course
uv init ragcourse && cd ragcourse
uv add requests rank-bm25 chromadb ollama

Verify: uv run python -c "import chromadb, rank_bm25; print('environment OK')"

Step 3 · Version control

git init
git add . && git commit -m "Week 1: environment setup"

Every checkpoint in this course ends in a commit. By Week 8 this repository is your portfolio.

Step 4 · First programmatic call

Ollama exposes a local HTTP API on localhost:11434 — the same request/response pattern as the OpenAI and Anthropic APIs, so everything you learn transfers. Create hello_llm.py:

import ollama

response = ollama.chat(
    model="llama3.1:8b",
    messages=[{"role": "user",
               "content": "What is retrieval-augmented generation, in two sentences?"}],
)
print(response["message"]["content"])

Run it: uv run python hello_llm.py — then commit.

Step 5 · Probe the knowledge boundary

Now we measure Part 1 instead of taking it on faith. Create probe.py:

import ollama

QUESTIONS = {
    "stable":    ["What year did World War II end?"],          # add 4 more
    "long_tail": ["Who founded the first bakery in Ushuaia?"], # add 4 more
    "post_cutoff": ["What happened in tech news last month?"], # add 4 more
    "capstone":  ["<a question only YOUR corpus can answer>"], # add 4 more
}

for category, questions in QUESTIONS.items():
    print(f"\n=== {category.upper()} ===")
    for q in questions:
        r = ollama.chat(model="llama3.1:8b",
                        messages=[{"role": "user", "content": q + " Answer briefly."}])
        print(f"\nQ: {q}\nA: {r['message']['content'][:300]}")

Fill in five questions per category, run it, and score every answer yourself: ✅ correct, ❌ wrong, 🤷 refused. In session we compare maps: the pattern — reliable on stable knowledge, degrading on long-tail, inventing on private — will be visible in everyone's results, on different questions. That shared pattern is the empirical case for this entire course.

Step 6 · Choose your capstone corpus

Pick the document collection your system will serve all eight weeks. Criteria:

• You can judge answers. You'll be the ground truth in evaluation weeks — pick a domain you know cold.
• 20+ documents, mostly text (PDF, markdown, HTML all fine). Enough that "just paste it all in" visibly fails.
• Real questions exist. Someone — you, colleagues, students — actually wants answers from this corpus.

Good picks from past cohorts: your own course materials, a product's documentation, a set of research papers, internal process docs.