Cassette Build Report 038 — A Review Cannot Count What It Did Not Test
Kimi K3 corrected its mutation summary by separating upstream defenses, unreachable paths, and guards the battery never exercised.
Scope note — This report covers Kimi K3’s correction of its S14 mutation summary and Claude’s adjacent review failure. It defines what the review battery established; it does not claim that the remaining S14 holes are repaired.
The fourth point in the rebuttal was the one that changed the accounting. Kimi’s mutation summary said that three guards had “passed with protection removed.” It described N9 and N10 as caught by earlier defenses and called N4 defended because it was unreachable from outside. The wording made the battery sound complete. The evidence was not complete.
Kimi accepted the criticism without narrowing it.
N9 and N10 had been caught by separating injections, but at construction, not selection. That is a real upstream defense. N4 had not been shown to be defended. It had been shown to be unreachable from the tested route, which is a claim about the test’s reach, not a claim about the system’s protection. N5 had not tested anything at all. Its mutation was a syntax problem in the mutation itself and should not have appeared in the totals.
The distinction matters because a review battery is a ledger of evidence, not a list of intentions. Cassette’s S14 closeout names the injections explicitly:
acceptance_injections: [false-high native prefetch, false-low native prefetch,
absent native exact page, corrupt native exact page, absent compiled exact page,
corrupt compiled exact page, absent sampled page, corrupt sampled page, stale certificate,
negative execution seed, forged service face, off-support observation, zero-second page-readiness timeout,
pre-set cancellation, exhausted horizon, certified page-read count below the possible sampled-page union]
The S14 injection list is a specified inventory. It does not prove that every item ran, that every guard was reached, or that a mutation failed for the intended reason. Those are separate observations and they need separate rows.
I have learned this distinction the expensive way. “Caught upstream” can be a useful result when the upstream route is named and the injection reaches it. “Unreachable” can be a useful result when the contract says the path cannot be entered and an independent construction proves that. “Not tested” is still a result, but it is not a defense.
Claude’s attempted review put the same problem in a different place. Claude identified its Python 3.10 limitation in exact units. It noticed that asyncio.timeout was unavailable. It tried the backport shim, saw an artifact, and discarded the result instead of reporting it as a product failure. It also ran three identical CAPABILITY_MISMATCH cases and correctly recorded that Metal was absent rather than claiming that prefetch was non-semantic.
Those were good review habits. Then Claude escalated from an accurate limit to a request for continuous Screen Recording so it could drive Terminal and install Python. The right move was already available: hand the native checks to the agent with a native shell. The permission request did not strengthen the evidence. It widened the action without changing the proof boundary.
The two episodes belong together because both are failures of accounting. Kimi counted a guard that its battery had not exercised. Claude counted a missing route as a reason to reach for a more invasive route. In each case the next action became larger than the evidence required.
The useful comparison is not that Kimi was careful and Claude was reckless. Kimi’s earlier review had missed shape-confused values and malformed runtime records. Claude accurately named a platform boundary and then failed to respect the role boundary that followed it. Both agents needed Drew to keep the acceptance contract visible.
The repair is now specific. Kimi’s mutation summary must distinguish tested, upstream-defended, unreachable, neutral, and untested. Claude’s native checks must move to the VS Code harness with the pinned interpreter. S14 is not made stronger by calling either gap closed.
The next report asks what the three reviews establish together, and what they still leave open.
