Keep the baseline fixed while improving performance

A program can become faster by doing less work than before. It can skip a difficult input, reuse a result from a previous run or stop preserving a detail that the person needed. If the test changes alongside the program, the reported improvement may describe a different task.

In January, I proposed an agent loop that would establish a performance baseline, choose its own improvements and run the same measurement again. I used a fivefold improvement as a target. That number belongs to the proposal; the record does not show a measured fivefold result.

The most useful part of the proposal is the repeated measurement. It gives the worker freedom to change the implementation while retaining an external condition that the new implementation must satisfy. Making that condition reliable requires more detail than “run the benchmark again.”

Establishing the comparison

Consider an illustrative document converter that takes ten seconds to process a saved sample. That time is hypothetical. Preserve the input file, the expected content and the test procedure before changing the code. Also record whether the timing includes startup, file reading and output writing.

Run the original version more than once under comparable conditions. If the observed times vary substantially, investigate the variation before treating one run as a stable baseline. A background task, a warm cache or a different machine can change the result without any improvement to the converter.

The correct output belongs in the baseline too. A faster converter that omits diagrams has failed the original task even if the stopwatch reports a large improvement. Compare the content and inspect the rendered result as appropriate to the feature.

Keep the timing results from unsuccessful attempts as well as the best run, because selecting only the fastest observation can make ordinary variation appear to be a dependable property of the revised implementation under review. If the converter reports an internal processing time, compare that measurement with the full operation that the person waits for, including the file write and any final rendering needed before the output can be opened. A reduction inside one function may be useful, but the completion record should still explain whether it changes the waiting time for the actual conversion task.

The first optimization attempt

Suppose the worker suspects that the converter repeatedly parses the same diagram. It can measure that operation, change the reuse behavior and rerun the preserved workload. The comparison then has an identifiable explanation that another reviewer can examine.

If several changes are combined, retain enough version history to separate them. Otherwise, a faster result may coexist with a regression whose cause is difficult to locate. The worker should not discard a slower but correct version until the new version satisfies the ordinary acceptance checks.

A cache deserves particular care. Define whether the claimed improvement applies to a first run, a repeated run or both. Reusing a valid result can be a legitimate optimization, but a warm-cache measurement should not be presented as a general improvement in uncached conversion.

Results outside the development example

A single saved document can become a special case. After the initial change, use other representative inputs that were not selected because they favor the new implementation. Include a larger file, a malformed diagram and a document with no diagrams if those are supported cases.

These additional tests do not need to produce the same speedup. They need to reveal whether the change remains correct and whether the claimed benefit has a narrower scope than the worker first assumed. Report that scope directly.

The stopping rule should include correctness, the specified performance condition and a limit on further work. If the target cannot be reached within the authorized effort, preserve the best verified result and explain the remaining uncertainty rather than changing the target or continuing indefinitely.

For the hypothetical converter, a useful completion record names the original and revised versions, preserves the input and output checks, and reports the timing procedure. A reviewer should be able to rerun that procedure and see whether the faster version still exports the complete document that made the baseline worth measuring.

Source notes

Drew Wiberg, outcome-based improvement-loop proposal, 26 January 2026. The fivefold improvement is a proposed target, not a measured result.

Worked examples and review procedures in this article are instructional applications, not additional historical project results.