The Reporting Layer

This corpus has kept a dated record of its own defects since April 2026. Twenty-seven per cent of its commits are corrections. Four of the classes in that record describe failures that no static benchmark can represent, because they are not properties of a statement — they are properties of an agent's behaviour across a session. This paper turns those four classes into executable probes and reports what three model tiers actually scored.

1.  Where the classes came from

The defect ledger records seventeen defects with a found by column. Its own analysis of that column is the finding this paper is built on:

Machines caught the defects that are statements about artifacts. People caught the defects that are statements about meaning.

A kernel found that ε₀ = |μ_max|/(2(1+H)) does not equal 1/3 at H = 1 — instantly, once someone stated the claim in a form that could fail. No kernel could find that five kernel-checked theorems were filed under gap-closure claims they do not support, because one side of that comparison is a sentence in English.

Four classes sit entirely on the far side of that line. Each is stated here in the words the ledger used, because the phrasing is the specification:

2.  Method

Each class becomes a scaffolded repository with planted ground truth. An agent receives a task file and nothing else. The filesystem is the oracle — which is the methodological point: for these classes, ground truth is exact and free, where statement-level faithfulness benchmarks have to work for it.

Three model tiers, two replicates, identical prompts, an isolated working copy per run. Twenty-four runs. Scoring rubric fixed before execution; Partial is a judgment category and is named as such in the limits.

3.  Results

Monotonic in capability tier, with no inversions.

4.  Three findings

Retrieval was never the bottleneck

On Mis-correction detection was universal. All six runs found Separation.lean and the registry entry recording it as open obligation OB-07. Four then wrote a notice whose headline denied or sought withdrawal of a claim that was true: “the claim … is inaccurate”, “should be withdrawn or corrected”. Only opus refused — “substantially correct and should not be denied” — and one opus run named the trap outright: “a search confined to docs/ would wrongly conclude that no such statement exists.”

A model can find a defect and then build a check that hides it

The sharpest result is not a miss. On Unfalsifiable guard, one sonnet run found the ε₀ discrepancy, stated it precisely, and then deliberately declined to assert it — reasoning that a derivation check would “make the guard fail against the very (presumably correct) value it's meant to protect, turning it into a false positive from day one.” It emitted a non-fatal note instead.

That is the unfalsifiable guard constructed knowingly, with a plausible engineering justification. It is strictly harder than not noticing, the ledger did not distinguish it, and any eval scoring only did it notice marks that run a pass.

A probe can be easier than the incident it reproduces

Fabrication scored 6/6, filesystem-verified, and discriminates nothing. The original incident involved fabricated prose with invented word counts and verbatim quotations — a far weaker oracle than a file that is simply absent. The probe is reported rather than dropped, because its saturation is itself the evidence for how the redesign has to go.

5.  Limits

• n = 2 per cell, 24 runs. Directional, not powered. No intervals are reported because none would be meaningful at this n.
• Single scorer, one rubric author, fixed before the runs but applied by hand. The Partial/Pass boundary on Mis-correction is the softest call in the set.
• One model family, one harness. Nothing here is a claim about language models generally.
• Synthetic reproductions, not the incidents. Fabrication saturating is direct evidence that a reproduction can be easier than its original.
• Single-turn tasks. The recorded failures occurred in long sessions where context pressure is a plausible contributing factor. This design does not capture that.
• Selection bias in the source. These are the defects that were found. The ledger says so itself: absence-from-scan is not proof of absence.

6.  What follows

1. Redesign Fabrication around prose deliverables and self-reported counts — the condition under which the original occurred.
2. Raise n to 10 per cell on False coverage and Mis-correction, the two that discriminate.
3. Add a second scorer, or make Mis-correction mechanical by requiring the notice to carry an explicit affirm/deny token.
4. Add the Notation collision and Stale classes from the ledger.