Arriving With a Corpus

A paid, cohort-based RAG course gives you eight weeks of expert attention. What you get out of it is decided almost entirely by something you control before day one: whether you arrive with a real corpus and a real question, or wait to be handed a tidy one.

Bring your own corpus

Every RAG curriculum needs a document collection to work on, and most supply one: a tidy set of clean articles where chunking is uncontroversial, metadata is uniform, and the questions have answers. You will learn the techniques. You will not meet the problems.

Real corpora are heterogeneous, contradictory, badly formatted, and partly wrong. They mix PDFs with HTML with spreadsheets. They contain two documents that disagree about the same number. They have a version-two that supersedes a version-one nobody deleted. Those are the problems that define the job, and you only meet them if the corpus is yours. MODEL

Good candidates: your firm's contracts or SOPs; a regulatory corpus in your field; your own writing across a decade; a public dataset in a domain where you are the expert. Bad candidates: Wikipedia, arXiv abstracts, anything already used in a tutorial — you will be comparing yourself to a solved problem.

Three hours that change the whole course

Not to get ahead. To arrive with questions rather than a blank page.

1 · A keyword baseline you wrote yourself

Implement BM25 — or use a stock implementation, but read it. Fifty lines. Then run twenty real questions through it and read what comes back. You will form opinions about your corpus that no lecture can give you, and every later technique gets measured against this.

2 · Twenty questions with known answers

Write them by hand, before you have a system to flatter. Include five you believe the corpus cannot answer — the refusal cases are the ones that separate an engineer from a demo-builder, and they are the first thing that gets skipped when a deadline arrives.

3 · One deliberate contradiction

Find, or plant, a case where two documents disagree about a fact. Keep it in a file. When the course reaches evaluation, ask your system that question and watch what it does. Most systems pick the higher-ranked passage and answer with confidence. That behaviour is the single most consequential defect in deployed RAG, and you want to have seen it in your own system, early. DATA

Ask these in every module

#	Question	Why it earns its place
Q1	What does this beat, and by how much, on my data?	Converts a technique from fashion into a measurement. If nobody can answer it, the technique is unproven, not wrong.
Q2	What does it cost — latency, dollars, complexity?	A reranker that adds 400 ms and 3% accuracy is a good trade in research and a bad one in a chat interface. The right answer depends on a budget nobody states aloud.
Q3	How does it fail, and will I notice?	Silent degradation is the dangerous kind. A system that returns nothing is debuggable; one that returns confident nonsense is not.
Q4	What happens when the source document changes?	Almost never covered, and it is most of the operational life of the system. See §5.

Ask them out loud. Cohort courses reward the student who makes the instructor defend a claim, and the rest of the cohort benefits from the answer. This is the part of a live programme you cannot get from a video. VALUE PREMISE

Framework defaults are somebody else's answer

chunk_size=512, overlap=50, k=5. These numbers appear in nearly every tutorial. They are not wrong; they are someone else's tuning on someone else's corpus, frozen into a default and repeated until it looked like knowledge.

Your corpus has a natural unit. Legal contracts have clauses. Research papers have claims. Transcripts have turns. Code has functions. Chunking on a token count cuts across every one of them — and the specific damage is that a qualifier gets separated from the statement it qualifies. Retrieve the sentence "the boundary is 0.776" without the adjacent sentence that says the value is not yet established, and your system has manufactured certainty out of a correctly-hedged source. MODEL

So run the experiment the tutorials skip: chunk your corpus three ways — fixed-size, structural, and one hybrid — and measure. It is an afternoon. It routinely matters more than the embedding model, and it is the cheapest lesson in the course.

Four things that decide whether the system survives

Provenance

Retrieval usually returns text. Retrieval should return text plus its status: where it came from, how confident the source was, whether it has been superseded. A corpus whose claims carry explicit status lets the generator be bound by a contract — never present an unestablished claim as established, never state a number absent from context, report conflicts rather than silently resolving them. Most corpora lack that metadata; adding even a crude version of it is high-leverage. MODEL

Invalidation

A source document changes. Which of your answers are now wrong? Systems that cannot answer this are search boxes with a language model attached. The fix is a dependency graph: claims depend on facts, facts on documents, so a change propagates a closure of things now stale. This is make, applied to knowledge, and it is the difference between a project and a system. MODEL

Refusal

The most valuable behaviour in production is declining to answer. It is also the least demoed, because a refusal makes a bad screenshot. Decide early what your system does when retrieval returns nothing relevant, when passages conflict, and when the question is outside the corpus. Then test those paths as deliberately as the happy one.

Operations

Who re-indexes when documents change? What breaks when the embedding model version rolls? What does the handoff look like when the person who built it is unavailable and someone else must fix it at 4pm on a Friday? None of this is glamorous and all of it is the job. DATA

Build the thing you will keep using

The strongest capstones are not the most technically ambitious. They are the ones whose author still runs them in month four. If your system answers a question you actually have, you will keep maintaining it, and maintenance is where the remaining 80% of the learning lives.

Two habits worth carrying out of the course:

• Keep a decision log. One line per design choice, with the number that justified it. In six months you will not remember why k=8, and neither will anyone reviewing your work. This is also, precisely, what makes a capstone defensible under questioning.
• Write the limits down. Every system has queries it handles badly. A README that names them is worth more to a reader than one that claims none exist — and a reviewer who finds an unlisted failure trusts everything else less. VALUE PREMISE

Before week one

• Corpus chosen; licence and permission settled
• Corpus in one folder, one format-per-type, actually parseable
• Twenty questions written by hand, five of them unanswerable
• One known contradiction identified and saved
• BM25 baseline running; twenty questions scored
• Git repository initialised; first commit is the baseline
• Local model runnable offline (Ollama or equivalent), so a network outage costs nothing
• Decision log file created — empty is fine, existing is the point

Eight items, one weekend. The student who arrives with these has already had the argument with their own data that the course is designed to provoke — and spends eight weeks going deeper instead of catching up.

A worked example you can read in a sitting

AXLE-RAG is a small retrieval system over a live research corpus: claim-level chunking, BM25 with provenance weighting, a refusal contract, a defect audit, and a dependency graph for invalidation. Roughly 1,100 lines, standard library only, no API key and no network. It is MIT-licensed and deliberately small enough to read rather than merely run.

It is offered as a second worked example, not a better one. Its most useful output is not an answer: run over its own corpus it reported one physical constant asserted at four different values across four files — a defect that had previously taken a full working session to find by hand. DATA

Reference implementation →