Agent-Generated Tests Do Not Prove Software Is Ready
Seven recent preprints find that coding agents often miss changed lines, error handling, stable assertions, acceptance requirements, and disciplined verification.
Scope note: This review covers seven recent studies of tests, verification, process discipline, and maintainability in autonomous software work. It does not evaluate one coding product or claim that agent-written tests are generally useless.
Coding agents can write tests. That fact tells us very little about whether the tests check the right behavior.
Seven recent preprints examine agent-generated tests and the processes around them. The results are mixed in a useful way. Agents often test more boundary conditions than people do. They also produce more unstable tests, leave large portions of changed code untested, and write tests that provide information without making a strong assertion.
My conclusion is not that agents should stop writing tests. It is that test presence should never be used as a substitute for software verification.
More tests can still provide weak evidence
Beyond Test Presence analyzes 204,673 test artifacts, including 179,732 generated by agents. Agent tests covered a wider range of boundary checks and used null-safety cases more often than human tests. Human tests held a small advantage in strong assertions, 88.1 percent compared with 85.37 percent. Agent tests also had a higher potential flakiness rate, 0.41 compared with 0.30, largely because they depended more on file operations and nondeterministic behavior.
This is not a simple quality loss. The agents found useful edge cases. The problem was that their tests were less aware of environmental conditions that affect repeatability.
Test Coverage Analysis of Agentic Pull Requests examines 4,882 agent-created pull requests. Agents changed tests in only 49.6 percent of relevant pull requests. Existing tests executed 61.5 percent of changed Java lines and 27 percent of changed Python lines. In Python, 64.8 percent of pull requests had no changed line executed by an existing test. Error handling was especially weak, with missed rates reaching 86 percent in Java and 81 percent in Python.
A green test command can therefore coexist with substantial untested change. The command proves that the tests passed. It does not prove that the modified behavior ran.
Agents often use tests as diagnostic output
Rethinking the Value of Agent-Generated Tests studies six models on SWE-bench Verified. Resolved and unresolved tasks had similar test-writing rates. The generated tests often behaved like diagnostic probes, with print statements used more frequently than formal assertions. Prompting agents to write more or fewer tests did not significantly change final outcomes.
That result explains behavior I see in practical coding work. A temporary test can help an agent inspect a value or confirm a hypothesis. It may be useful during development and still be unsuitable for the permanent test suite.
Agentic Rubrics offers another form of verification. An expert agent inspects the repository and creates a task-specific checklist, then evaluates proposed patches against it without executing the code. The approach improved its tested baselines by at least 3.5 percentage points and identified issues that the ground-truth tests did not capture.
Rubrics do not replace execution. They can check requirements that an existing test suite does not express: compatibility, repository conventions, data handling, or an explicit user constraint.
Process discipline affects code quality
RigorBench measures planning, verification coverage, recovery, appropriate refusal, and the integrity of individual changes. Structured agent processes improved process scores by an average of 41 percent and outcome correctness by 17 percent in the reported study.
TRIM analyzes unnecessary edits left behind during an agent’s search for a passing solution. Its method uses the trajectory to identify and remove changes that are not required for the final behavior. It reduced unnecessary code by 17.9 to 32.9 percent across the tested agent systems with little performance loss.
The maintenance concern is direct. An agent can satisfy the current test while leaving speculative helpers, duplicate branches, or abandoned changes in the repository. Those changes increase future reading and testing work even when the immediate behavior is correct.
Acceptance tests must exercise the actual product
From Runnable to Shippable studies full web applications. Its TDDev system converts requirements into acceptance tests before implementation, deploys the application, operates it through a browser, and turns observed failures into repair instructions. The process improved generation quality by 34 to 48 percentage points over a no-TDD baseline.
The study also found that the best protocol depended on how the model generated software. A mismatched protocol removed the benefit and increased token cost by as much as 25 times. This is a useful warning against treating one agent workflow as universal.
What I would accept
For agent-written code, I want several kinds of evidence. Existing tests should pass. Changed lines should be measured for execution. Error and recovery paths should receive direct cases. New tests should contain meaningful assertions and remain stable across repeated runs. User-facing behavior should be exercised through the real interface. The final diff should be reviewed for changes that do not contribute to the requirement.
I also want the agent to state what it did not verify. That statement lets a reviewer direct attention where evidence is missing.
Agent-generated tests are useful work products. They are not independent proof because the same system can write the implementation, choose the test, and interpret the result. Software readiness requires several checks with different failure conditions. The research now gives us enough evidence to make that standard explicit.