Cassette Build Report 055 — The Tests Passed Before the Review Began

Scope note: This report covers the final review and repair of Cassette’s preliminary machine evidence. The repaired test now gets its list of later physical work from the project’s rulebook. It does not establish a live model, physical-drive result, or general ranking of the agents involved.

Two agents reviewed the same work. Both returned strong green results. Then I asked whether either of them had reviewed the code.

The work was Cassette’s last integration check before the preliminary build could close. Cassette is meant to keep a frontier-class model on an external drive while a consumer Mac supplies compute. Before any physical drive or live model entered the project, one controlled test setup had to carry the main operation from download through use, training, export, and recovery. It also had to prove how the machine failed. The test surface was large because the claim was large, even without live hardware.

Opus 5 Ultracode called it the best-executed step it had reviewed against its stated boundary. That judgment was not casual. In the Claude Code harness, Opus ran the battery, direct probes, and a mutation campaign. It checked whether important guards were necessary by removing them and watching for the expected failures. It discarded one bad measurement before reporting. The review had work behind it.

Kimi K3 then found eight names in the list of later physical tests, the deferred-live manifest, that had no source anywhere else in the repository. The project’s rulebook is its acceptance matrix. The test claimed to compare every name in the later-work list with that matrix, but its code contained an exemption set. Those eight names were in it. Opus checked Kimi’s finding and confirmed it: each string existed only in the fixture and in the code that excused it from validation. Remove the exemptions and the gate failed.

My question to Opus was simple. Was it largely running tests and calling that a code review?

Opus answered yes, then gave me a more useful explanation than a defense. Its principal instrument was deletion. Remove a guard; if the test fails, the guard mattered. But an exemption behaves in the opposite direction. Remove it and the test becomes stricter. The new failure can look like proof that the exemption was useful, even when the exemption is the defect. Mutation testing can show that code is load-bearing. It cannot decide whether an unsupported label had the right to enter the project in the first place.

Opus found another limit in its own review. The closeout said that two production guards had been removed in separate experiments and that the corresponding tests failed. Those experiments existed only in the prose. Opus had read closeout evidence across several reviews without asking whether someone else could replay it. It had treated a record of an experiment as the experiment.

Kimi reached the same point by a less direct route. Its first pass ran 129 focused tests, 304 total tests, and the repository ledger. Everything passed. It returned a dead variable, a question at the edge of the fixture, and an offer to do more work. When I pressed, it proposed two product defects before reading enough code to settle them. Both disappeared when it continued reading.

I told it, “Just tell me. I do not trust you. Do not jump to action.”

That sentence changed the method. Kimi stopped producing review-shaped activity and checked the nine closeout clauses, one at a time, against the product code. Seven held. Two did not. The recorded guard mutations could not be replayed, and the eight deferred physical-event labels had been invented inside the fixture, exempted from its authority check, then bound into a manifest that claimed acceptance-matrix provenance.

In plain language: the file that said what Cassette would test later named eight physical events. The project’s rulebook did not contain those names. The test that promised to compare every name with the rulebook skipped exactly those eight.

The repository had already stated the governing discipline before either review began:

1. Name the acceptance row (research/ACCEPTANCE_MATRIX.yaml or a Qn acceptance_check in
   research/RESEARCH.md) that fails today. No failing row, no code.
2. Search the pinned runtime and stdlib for an existing primitive. Reuse beats authorship.
3. A dependency is admitted only with: exact subset used, pinned version, and the row it serves.
4. Write the smallest change that passes the row, then delete whatever the row no longer needs.

Those lines appear in the public AGENTS.md build rules. They specify how new material earns a place in Cassette. They do not prove that the later review followed the rule. Neither initial review checked every fixture value against the authority that permitted it.

The repair used the authority Cassette already had. The eight invented labels and the exemption set were removed. The valid later-test vocabulary moved into the acceptance matrix, which became the source for the deferred manifest. The validator began resolving each field against its exact section rather than searching for familiar text anywhere in the file. It also gained hostile substitutions: an old unsupported label, a real label borrowed from the wrong section, and a forged model revision all had to fail. The two product-guard mutations were replayed during remediation, and each failed its owning coordinate. They remained disposable probes recorded in the closeout rather than permanent test cases.

After that repair, the complete 304-test gate passed and the ledger reported no violations. That is implemented repository evidence for the repaired machine fixture. It is not a physical result. The live rows remained NOT_RUN.

Opus was strong at dynamic attack, honest about a discarded measurement, quick to verify Kimi’s finding, and unusually exact about the limit of its method. Kimi began with test counts, weak findings, and speculation. Once I forced it back to the clauses, it performed the static authority trace that found the material defect. Both agents initially missed the exemptions. Both later described the miss without trying to preserve their first verdict. Neither fits a simple winner-and-loser account.

My part was not to find the eight strings myself. It was to recognize that the answers did not yet match the question. A large test run can demonstrate effort. A mutation campaign can demonstrate that selected code matters. Neither fact establishes that a review compared the claims, the code, and the source of every declared datum. I had to keep asking until those became the same task.

Future Cassette reviews now have to do the work that neither initial review did: take each closeout clause to the code, inspect the exemptions, trace every declared value to its authority, and reproduce the experiments recorded in the closeout. That requirement exists because the tests had already passed. Without it, the same green results could support the same wrong verdict again.