Cassette Build Report 042 — The Independent Oracle Proved the Wrong Transformer
Kimi's separate numerical oracle was correct about the executed graph, while Opus showed that the graph itself was not the transformer S15 had promised.
Scope note: This report covers the S15 review and repair of the diagnostic transformer stage. It distinguishes numerical independence from semantic correctness; it does not claim a general transformer compiler, frontier-scale model quality, or production context growth.
Kimi K3 Max gave S15 an independent oracle. It did not import the fixture’s expected-output helper. It rebuilt the arithmetic and matched the executed graph to the digit. That was good evidence. It was not enough.
Opus 5 Max read the same step and asked a prior question: what graph did the acceptance clause actually require? S15 was supposed to be a small causal transformer with recurrent key-value state. The executed path had an embedding, three projections, one attention head, and four values called logits. The vocabulary happened to be four wide because the attention output was four values wide. Shape supplied the name.
Kimi’s oracle proved the implementation it was given. The implementation was not yet the graph the stage promised.
The repair makes the graph an explicit generated sequence:
_TRANSFORMER_OPERATOR_KINDS = (
"embedding", "norm", "matmul", "matmul", "matmul", "rope",
"attention", "matmul", "add", "norm", "matmul", "silu",
"matmul", "add", "norm", "matmul",
)
The protected operator sequence in pager.py is implemented evidence for the repaired S15 boundary. The runtime rejects a graph whose generated operators do not appear exactly once in the declared order. It does not prove that every future model has this graph. It makes this diagnostic graph impossible to abbreviate silently.
The missing operations were not a matter of style. There was no normalization around the attention path, no rotary position encoding in the executed trace, no feed-forward block, no residual structure, no output projection, and no unembedding. RoPE and RMS norm existed in a dispatch table and did not appear in the end-to-end trace. The stage could not test the properties it claimed to isolate.
The second finding was about sampling. Only the value projection was certified while query and key remained fixed. The sampled quantity therefore entered the output through a linear path, so the weighted average of sampled outputs equaled the output of the weighted average as an arithmetic identity. Agreement at 1e-12 proved the linear calculation. It did not test what happens after softmax and the feed-forward nonlinearity, where the same averaging claim is not automatic.
Kimi later accepted the correction. It also accepted that its key-value probe had not isolated cache consumption: changing the prefill history changed the input, so it did not prove that the committed cache was being used. Claude’s surviving mutation made the point operationally. Remove the cache and the fixture still passed.
This exchange changed what I mean by an independent oracle. It must calculate separately, but it must also be aimed at the declared object. A separate calculator can remove fixture circularity while preserving a semantic mistake. The oracle needs a name for the graph, the stage needs an assertion for the graph, and the mutation needs to remove the exact boundary that the assertion owns.
Sol rebuilt S15 around that distinction. The decoder now runs the full protected nonlinear graph, carries the committed key-value state into one-token decode, and checks the observed execution error and risk against the certificate. The repair was not a larger test list attached to the old path. It changed the path so that the stage’s question could become false.
The comparison is useful because nobody needed to be declared the winner. Kimi found the arithmetic weakness of a self-proving fixture. Opus found the semantic mismatch that the independent arithmetic inherited. Sol made the declared graph and its recurrence load-bearing. Drew kept the question from shrinking to whichever answer was easiest to reproduce.
An oracle can be independent and still be aimed at the wrong thing. S15 became credible only when both halves were checked: the numbers and the object those numbers were supposed to describe.
