Semantic Retrieval: Embeddings and Vector Search

Last week you caught BM25 failing on synonyms and paraphrase — words that mean the same thing but share no letters. This week's tool attacks exactly that gap: meaning as geometry. And you'll test your Challenge 2 predictions against reality.

Part 1 — Embeddings: meaning as coordinates

An embedding model is a neural network that maps a piece of text to a list of numbers — a point in a high-dimensional space (768 dimensions for the model we'll use). The training objective forces one property: texts with similar meaning land near each other. "Car" and "automobile" sit close together; "car" and "carpet" don't, despite sharing more letters.

Where BM25 asks "do the same words appear?", embedding search asks "do these texts point in the same semantic direction?" The standard measure is cosine similarity — the angle between the two vectors: 1.0 for identical direction, near 0 for unrelated. Search becomes: embed the query, embed every chunk (once, in advance), return the chunks with the smallest angle.

import ollama, math

def emb(text):
    return ollama.embed(model="nomic-embed-text", input=text)["embeddings"][0]

def cos(a, b):
    dot = sum(x*y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x*x for x in a)) * math.sqrt(sum(y*y for y in b)))

pairs = [("car", "automobile"), ("car", "carpet"), ("refund policy", "getting my money back")]
for a, b in pairs:
    print(f"{a!r} vs {b!r}: {cos(emb(a), emb(b)):.3f}")

Part 2 — Making it fast: approximate nearest neighbors

Exact search compares the query against every stored vector — O(N) per query, the same wall the inverted index solved for keywords. The vector answer is ANN indexes: data structures that find almost certainly the nearest vectors while examining a tiny fraction of them.

The dominant one, HNSW (Hierarchical Navigable Small World), builds a layered graph — sparse express layers on top, dense local layers below. A query enters at the top, greedily hops toward its target, and descends. Think: fly to the right country, drive to the right city, walk to the right door. The price of the speed is the word approximate: recall of ~0.95–0.99 against exact search, tunable. For our corpus sizes exactness is cheap; for millions of vectors, ANN is the only game — and knowing the trade-off exists is what matters at design reviews.

Part 3 — Chunking: the decision that quietly dominates quality

You can't embed a 40-page document as one vector — its meaning averages into mush, and you couldn't fit it in the context window anyway. Documents must be split into chunks, and this unglamorous choice routinely moves retrieval quality more than switching embedding models:

The core tension: small chunks retrieve precisely but may lack context to answer with; big chunks carry context but blur into noise and eat the token budget. There is no universal answer — which is why the lab measures, on your corpus, instead of copying a tutorial's chunk_size=1000.

Part 4 — Vector databases: indexes plus bookkeeping

A vector DB (we'll use Chroma — embedded, zero-config, pure local) stores chunk texts, their vectors, and metadata; serves ANN queries; and handles persistence and filtering. That's it. The category is heavily marketed, so keep the deflationary view: a vector database is an ANN index with bookkeeping. Postgres with pgvector, Qdrant, or a NumPy array all occupy the same seat at different scales. Choosing one is a Week 8 ops question, not a Week 3 intelligence question.

Lab — semantic search over your corpus, three chunkings, verdict by numbers

Step 1 · Chunkers

Create chunkers.py:

def fixed(text, size=800, overlap=150):
    chunks, i = [], 0
    while i < len(text):
        chunks.append(text[i:i+size])
        i += size - overlap
    return chunks

def paragraphs(text, max_len=1200):
    paras, out, cur = [p.strip() for p in text.split("\n\n") if p.strip()], [], ""
    for p in paras:
        if len(cur) + len(p) < max_len:
            cur = cur + "\n\n" + p if cur else p
        else:
            if cur: out.append(cur)
            cur = p
    if cur: out.append(cur)
    return out

def titled(text, max_len=1200):
    """Paragraph chunks, but every chunk carries the last seen heading."""
    title, out = "", []
    for chunk in paragraphs(text, max_len):
        first = chunk.splitlines()[0]
        if first.startswith("#"): title = first.lstrip("# ")
        out.append((f"[{title}] " if title else "") + chunk)
    return out

Step 2 · Embed and index

Create vector_index.py. One Chroma collection per chunking strategy, so they compete side by side:

from pathlib import Path
import ollama, chromadb
import chunkers

client = chromadb.PersistentClient(path="chroma_db")

def embed(texts):
    return ollama.embed(model="nomic-embed-text", input=texts)["embeddings"]

def build(strategy_name, chunk_fn):
    col = client.get_or_create_collection(strategy_name)
    if col.count(): return col                    # already built
    for p in Path("corpus").glob("*"):
        if p.suffix not in (".txt", ".md"): continue
        chunks = chunk_fn(p.read_text(errors="ignore"))
        col.add(ids=[f"{p.name}::{i}" for i in range(len(chunks))],
                documents=chunks,
                embeddings=embed(chunks),
                metadatas=[{"doc": p.name}] * len(chunks))
    print(f"{strategy_name}: {col.count()} chunks")
    return col

def search(col, query, k=5):
    """Chunk-level hits, mapped back to parent documents for fair comparison
    with the Week 2 baseline (which ranks whole documents)."""
    res = col.query(query_embeddings=embed([query]), n_results=k*3)
    docs, seen = [], set()
    for m in res["metadatas"][0]:
        if m["doc"] not in seen:
            seen.add(m["doc"]); docs.append(m["doc"])
        if len(docs) == k: break
    return docs

STRATEGIES = {"fixed": chunkers.fixed, "paras": chunkers.paragraphs, "titled": chunkers.titled}

if __name__ == "__main__":
    for name, fn in STRATEGIES.items():
        build(name, fn)

Note the mapping step: your golden queries label documents, but this index retrieves chunks. Mapping chunk hits to parent docs keeps the comparison honest. (From Week 6 on, chunks themselves go into the context window — this is just for like-for-like scoring.)

Step 3 · The bake-off

Create evaluate_semantic.py, reusing your golden queries:

import json
import vector_index as vi
from bm25_baseline import search as bm25_search

queries = json.load(open("eval_queries.json"))
K = 5

def score(get_top):
    recalls, rrs = [], []
    for item in queries:
        top, rel = get_top(item["query"]), set(item["relevant"])
        recalls.append(len(rel & set(top)) / len(rel))
        rrs.append(next((1/r for r, d in enumerate(top, 1) if d in rel), 0.0))
    return sum(recalls)/len(recalls), sum(rrs)/len(rrs)

print(f"{'system':<12}{'Recall@5':>10}{'MRR':>8}")
r, m = score(lambda q: [d for d, _ in bm25_search(q, k=K)])
print(f"{'bm25':<12}{r:>10.3f}{m:>8.3f}")
for name in vi.STRATEGIES:
    col = vi.client.get_collection(name)
    r, m = score(lambda q, c=col: vi.search(c, q, k=K))
    print(f"{name:<12}{r:>10.3f}{m:>8.3f}")

Run it. One table, four systems, your corpus. Record the winner in your README — and don't be shocked if BM25 still wins some rows. That result is normal, it's corpus-dependent, and it's exactly why Week 4 fuses the two instead of picking a religion.

Step 4 · Judgment day for Challenge 2

Run your five "Break BM25" queries through the best semantic collection. For each: did embeddings fix it, as you predicted? Write the scorecard into bm25_failures.md — predictions vs. reality. Typical pattern: synonym and paraphrase failures fixed; exact identifiers (course codes, names, error strings) now worse, because embeddings blur precisely what keywords nail. Keep that asymmetry in mind all next week.