The Machines That Read Us · Book XIV

This corpus reads itself with four small programs. One finds file names, one files pages under sections, one links pages that share ground, and one collects the series' own keys. A standard textbook of language processing has a name for each of them, and a way to measure each. This chapter measures them the way the textbook would.

1 · The four readers

Every tool here is a textbook technique

tool	technique	where J&M teach it	what it does here
tools/lean_addresses.py	regular expressions and tokenization	J&M p.33	read CSS selectors as file names (nine false hits) and then its sibling tool's “FILE.lean” on 96 pages (XIX ch 4)
tools/subject_tags.py	text categorization	J&M p.92	labels are assigned by an editor, not learned; §2 tests whether they could be
tools/crossref.py	tf-idf vectors and cosine similarity	J&M p.135	“same ground” suggestions above a cosine floor of 0.15
tools/key_register.py	lexico-syntactic patterns (Hearst)	J&M p.476	“K: threshold” is a pattern; a course legend matched it once in a sample of five (ch 4)

Jurafsky and Martin: "All tokenization systems also depend on regular expressions as a processing step" (p.33). Text categorization is "the task of assigning a label or category to a text or document" (p.92). "The dot product acts as a similarity metric" (p.135). And "the earliest and still common algorithm for relation extraction is lexico-syntactic patterns" (p.476). CITED

What the corpus's record showsEvery one of these tools first failed on precision: it reported things that were not there, such as CSS read as file names, a download box read as a citation, or a legend read as an operator. Each failure was found and fixed. Recall, the things a tool silently misses, was never measured, because measuring it needs a gold standard that nobody has built. COMPUTED

2 · An experiment

Could a machine have filed the pages?

An editor gave every chapter a subject tag (MATHEMATICS · PURE, BIOLOGY · SYSTEMS…). The script asks whether the page text alone predicts the field. It takes 472 tagged pages in the 12 fields that have at least ten pages each, splits them ten ways, trains on nine parts and tests on the tenth. COMPUTED

reader	accuracy
always answer “mathematics” (the majority baseline)	0.453
naive Bayes over word counts	0.663
tf-idf cosine, nearest neighbour (the method crossref.py uses)	0.684

Both clearly beat the baseline, so the sections are partly written into the vocabulary. But J&M warn about exactly this situation. A classifier that "stupidly classified every tweet" as the majority class scores 99.99% on a skewed set (p.113), which is why they report F1 per class. Per field, naive Bayes gives: CITED

field	pages	precision	recall	F1
mathematics	214	0.63	0.93	0.75
physics	70	0.63	0.60	0.61
biology	44	0.79	0.61	0.69
economics	30	0.76	0.83	0.79
education	27	0.80	0.15	0.25
series apparatus	16	1.00	0.12	0.22
earth	15	1.00	0.60	0.75
method	12	0.00	0.00	0.00
history	12	0.00	0.00	0.00
medicine	11	0.67	0.18	0.29
philosophy	11	0.00	0.00	0.00
chemistry	10	1.00	0.20	0.33

The macro-F1 is 0.391. Mathematics has recall 0.93 and precision 0.63: the classifier files almost anything mathematical-looking under mathematics, and 28 physics pages go there. Education, series apparatus, method, history and philosophy are almost never recovered. COMPUTED

What that says about the tagsThe fields the machine cannot find are the ones defined by purpose rather than vocabulary. A teaching page about the Lorenz system uses the words of physics, and a method page uses the words of whatever it audits. So the magazine's sections carry information the text does not: they record why a page exists, not only what it is about. That is an argument for keeping the tags editorial, as R25 already does. MODEL

3 · The patterns

The key register (Key Register) is a Hearst-style pattern reader: "K: threshold" plays the role of "NP such as NP" (p.476). In the by-eye sample of Chapter 4, 19 of 20 hits were true glosses, so precision is high on a small sample. Recall is unknown, because glosses written as "K, the threshold," or in prose are not caught. OPEN

Not establishedThe tags are an editor's decisions, not facts about the pages, so "accuracy" here means agreement with those decisions. One deterministic ten-fold split was used, with no stemming and no stop list, so a better model would score higher. Recall for any of the four tools needs a hand-built gold set, which does not exist yet. WANTED

See also Chapter 3 (a sense test), Chapter 7 (a 1985 claim that AI was reinventing Sanskrit grammar) and Book XIX ch 4 (the address tool's own failures).

D. Jurafsky & J. H. Martin, Speech and Language Processing, 3rd ed. draft of 19 August 2026 (printed page = PDF − 8). Held; hash in docs/floor-texts.tsv. The 2nd edition (2008 printing, marked 2007) is also held.