Hybrid Retrieval and Reranking

Week 3 ended on an asymmetry: embeddings fixed your paraphrase failures but blurred exact identifiers that BM25 nailed. Neither system wins alone — real query traffic contains both kinds. This week you stop picking a religion and build the architecture production teams actually run: fuse both, then let a heavier model re-order the shortlist. The win condition is numeric: hybrid must beat both parents.

Part 1 — Why fusion beats either parent

Lay your Week 3 bake-off table next to your Challenge 2 failure list and the pattern is already there:

The two systems fail on different queries. That's the precondition for fusion to work: combine two rankers whose errors are uncorrelated and the union covers both blind spots. This is the same logic as ensemble methods everywhere in ML — and it's why "BM25 + vectors + fusion" is the default retrieval stack at most serious shops.

Part 2 — Reciprocal Rank Fusion: embarrassingly simple, annoyingly hard to beat

Problem: BM25 scores live on one scale (unbounded, corpus-dependent), cosine similarities on another (0–1-ish). Averaging them is meaningless. You could normalize — min-max, z-scores — but every normalization scheme has pathological cases.

RRF sidesteps scores entirely and uses only positions:

RRF(doc) = Σ over rankers  1 / (k + rank_in_that_ranker)      # k = 60 by convention

A doc ranked 1st by BM25 and 3rd by vectors gets 1/61 + 1/63. A doc only one ranker found still scores — just less. The constant k=60 damps the difference between rank 1 and rank 5, so one ranker's confidence can't steamroll the other. Ranked #1 in both → top of the fused list; found by only one → still in the running.

Why it's beloved in production: no tuning, no normalization, robust to adding a third or fourth ranker later, and consistently within a hair of much fancier learned fusion. Twelve lines of code — you'll write them today.

Part 3 — Reranking: spend compute where it counts

Your embedding search is a bi-encoder: query and document embedded separately, meeting only at a cosine comparison. Fast — documents are pre-embedded — but shallow: the model can't see how query words interact with document words.

A cross-encoder reads query and document together, attention flowing between them, and outputs one relevance score. Far more accurate; far too slow to run against a whole corpus (it can't precompute anything — every query-document pair is a fresh forward pass).

The production pattern resolves the tension with a funnel:

corpus (thousands)
  → cheap retrieval: BM25 + vectors + RRF     → top ~20 candidates
  → expensive rerank: cross-encoder            → top 5 enter the context window

Cheap-and-broad feeds expensive-and-narrow. Recall is decided by the first stage (a doc missed there is gone forever); precision at the top is decided by the second. Typical lift: 5–15 points of ranking quality for ~100–300 ms. Whether that trade is worth it for your users is a measurement, not an opinion — and you'll make it today.

Part 4 — Query understanding: fix the query before blaming the index

Some failures aren't the retriever's fault — the query itself is broken: too short, ambiguous, or phrased in vocabulary the corpus never uses. Three practical repairs, all using the LLM you already run:

• Rewriting: ask the LLM to restate the query clearly ("how do i get my $ back" → "refund request process"). Cheap, often surprisingly effective.
• Expansion: generate synonyms/related terms and append them — a modern, learned version of what search engines did for decades.
• HyDE: ask the LLM to write a hypothetical answer, then embed that instead of the question. Answers live nearer to answers than questions do in embedding space — a genuinely clever trick that sometimes wins big and sometimes flops. Measure.

Lab — the staged pipeline, every stage earning its keep

Step 1 · RRF fusion

Create hybrid.py. It fuses your two existing retrievers at the document level:

from bm25_baseline import search as bm25_search
import vector_index as vi

BEST = "titled"                      # your winning collection from Week 3

def rrf(ranked_lists, k=60, top=5):
    scores = {}
    for lst in ranked_lists:
        for rank, doc in enumerate(lst, 1):
            scores[doc] = scores.get(doc, 0.0) + 1.0 / (k + rank)
    return [d for d, _ in sorted(scores.items(), key=lambda x: -x[1])][:top]

def hybrid_search(query, k=5, depth=20):
    bm25_docs = [d for d, _ in bm25_search(query, k=depth)]
    vec_docs = vi.search(vi.client.get_collection(BEST), query, k=depth)
    return rrf([bm25_docs, vec_docs], top=k)

Note depth=20: fusion needs to see past the top-5 of each parent — a doc ranked 8th by both is exactly the kind hybrid rescues.

Step 2 · Cross-encoder reranking

Add the dependency: uv add sentence-transformers (first run downloads the model, ~90 MB). Create rerank.py:

from pathlib import Path
from sentence_transformers import CrossEncoder
from hybrid import hybrid_search

model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
TEXTS = {p.name: p.read_text(errors="ignore")[:1500]
         for p in Path("corpus").glob("*") if p.suffix in (".txt", ".md")}

def reranked_search(query, k=5, depth=20):
    candidates = hybrid_search(query, k=depth, depth=depth)
    pairs = [(query, TEXTS[d]) for d in candidates]
    scores = model.predict(pairs)
    ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])
    return [d for d, _ in ranked[:k]]

(We rerank each document's first 1500 characters — a deliberate simplification; from Week 6 you'll rerank the actual retrieved chunks. Say this limitation out loud in session: knowing what your code approximates is half of engineering.)

Step 3 · The staged table — your win condition

Extend evaluate_semantic.py to score four systems on your golden queries:

from hybrid import hybrid_search
from rerank import reranked_search

# ... same score() helper as Week 3 ...
# print rows for: bm25 | best-vector | hybrid | hybrid+rerank

Read the table with Week 4's rules:

• Hybrid beats both parents? Win condition met. If not — don't shrug, diagnose: print per-query results and find which queries fusion hurt. Usually a depth problem or one parent flooding the list.
• Rerank beats hybrid? Then decide, like an engineer: is the lift worth ~200 ms and a new model dependency for your capstone's users? Either answer is fine; unjustified answers are not.

Step 4 · Query rewriting on your worst queries

Take your 3 worst golden queries by RR. Create rewrite.py:

import ollama
from hybrid import hybrid_search

def rewrite(query):
    r = ollama.chat(model="llama3.1:8b", messages=[{
        "role": "user",
        "content": f"Rewrite this search query to be clear and specific, using likely "
                   f"document vocabulary. Reply with ONLY the rewritten query.\n\n{query}"}])
    return r["message"]["content"].strip()

for q in ["<worst query 1>", "<worst query 2>", "<worst query 3>"]:
    rq = rewrite(q)
    print(f"\noriginal:  {q}\nrewritten: {rq}")
    print("  original top:", hybrid_search(q, k=3))
    print("  rewritten top:", hybrid_search(rq, k=3))

Sometimes the rewrite rescues the query; sometimes the LLM "clarifies" it into something you didn't ask. Both outcomes belong in your notes — Week 7's agent will make this rewrite decision automatically, and today you're learning when to trust it.