Turn a failure record into a tool specification
Extract the repeatable check, its inputs, and the evidence required to trust its result.
A useful refactor log contains more than a list of changed files. It may show which condition was inspected, what failed, which edit changed the outcome, and which check was run afterward. That sequence can support a reusable tool, provided the tool preserves the conditions that made the original check meaningful.
In February 2026, I reported turning the logs from a security refactor into a security tool. The surviving conversation records that report; it does not independently validate the tool’s implementation or security coverage. The useful method to examine is the conversion itself. How does a record of one repair become a procedure that can handle another repository without pretending the repositories are identical?
Begin by reading the log as an account of a particular run. Identify the original problem, the inspected files, the command that exposed it, the change applied, and the observation that supported completion. Preserve uncertainty where the record is incomplete. A confident completion message without the underlying command output cannot supply a missing test receipt.
Next, identify which parts can vary, separating the repository path supplied as an input from the language-specific manifest that determines which checker runs, while retaining the expected policy rule and installed tool version as explicit conditions of the execution. The original filename should remain in a regression fixture rather than becoming an accidental requirement for every future run.
Here is an illustrative conversion. A repair log shows that a configuration loader accepts an empty required value. The developer adds validation and confirms that a small failing fixture now produces an explicit error. A reusable check should accept the configuration schema and test fixture, run the loader under controlled conditions, and report whether the expected rejection occurred.
That tool has a narrow claim. It tests a specified validation behavior. It cannot infer that the application is secure merely because the behavior passes. Its output should name the checked rule, input revision, executed command, and observed result, leaving unrelated questions available for separate checks.
Retain a fixture that demonstrates the original defect before adding a repaired fixture, and run the proposed check against both versions before packaging it, because a check that passes both has failed to capture the distinction that justified the refactor. This is an inexpensive way to catch a familiar mistake in tool building, where a command runs successfully but never exercises the condition it was meant to inspect.
Then vary one relevant input. For the configuration example, test an absent value, an empty value, and a valid value because each can take a different code path. Include an unsupported schema version. The tool should report that limitation explicitly rather than converting its inability to inspect the input into a passing result.
A shell command returning zero is only useful when the tool defines what zero means. Some programs return success after skipping files. Others inspect only the current directory or exclude generated content. Preserve the coverage information with the result, especially when a report will be consumed by another agent that cannot see the terminal session.
The original refactor may have required permission to edit several files. A reusable diagnostic tool need not inherit that permission. Start with read-only inspection, make any repair operation separate, and state the files that operation can change. A tool extracted from a successful intervention should still stop when its inputs fall outside the intervention it knows how to verify.
This separation also makes review easier. A developer can adopt the diagnostic step without accepting every proposed repair. The tool can return a patch for inspection, while the calling workflow decides whether the current task authorizes applying it. Keep credentials and private source material out of fixtures prepared for distribution.
Before reuse, run the tool against a fresh disposable example whose layout differs from the original repository, checking its failure output as carefully as its passing output so that the next person can distinguish an unsupported project from an actual defect in the inspected code. A wrong working directory, missing dependency, or unreadable fixture should produce a distinct execution error with enough information to correct the setup.
The deliverable is a small package containing the procedure, supported inputs, negative and repaired fixtures, and inspectable receipts. Keep the original episode linked in its development records so that a later maintainer can see which failure the check was designed to catch. Expand its advertised coverage only when another test establishes the added behavior.
Source notes
Based on my February 2026 report of converting security-refactor logs into a tool. The configuration-loader example is illustrative and does not describe an independently validated security product.
