Computer-Use Agents Need Repeated Reliability Tests
Seven recent preprints show that a successful computer-use demo says little about repeated execution, recovery, process correctness, or professional workflows.
Scope note: This review covers repeated execution, process checks, recovery, and long professional workflows in seven recent computer-use studies. It does not compare every current commercial computer-use agent.
A computer-use agent completes a task in a recorded demonstration. The result looks decisive. The same agent receives the same task again and fails.
That second run contains more useful product information than the first.
Seven recent preprints examine computer-use reliability beyond one successful result. They test repeated runs, equivalent instructions, tool faults, process evidence, recovery behavior, interaction methods, and professional workflows. Together they show why a pass rate alone is not enough for production decisions.
Success can vary across identical runs
On the Reliability of Computer Use Agents repeats the same OSWorld tasks and studies three causes of variation: randomness during execution, ambiguity in the task, and changes in agent behavior. The authors found that reliability depended both on how the task was stated and whether the agent chose a stable execution strategy.
The practical implication is immediate. If an agent succeeds once in five attempts, a demonstration can present the success while ordinary use experiences the other four results. Evaluation should report repeated success, not only whether success occurred at least once.
ReliabilityBench formalizes this requirement. It measures consistency across repeated runs, robustness to equivalent task wording, and tolerance of tool failures such as timeouts, rate limits, partial responses, and schema changes. Across 1,280 episodes, modest wording changes reduced success from 96.9 to 88.1 percent. Rate limits caused the most damage among the tested faults.
These are normal production conditions. They should appear in acceptance tests.
The final screen can hide a bad process
ProBench adds process-related mobile tasks and a provider that records exact intermediate state. The authors show that final-screen inspection misses important failures because some required actions leave no visible evidence at the end.
This issue applies to consequential work. A file can appear in the correct folder even if the agent copied the wrong version and later renamed it. A form can display a confirmation page even if an optional but necessary field was skipped. A final state is useful evidence, but it cannot prove every required action occurred.
When Agents Fail to Act analyzes 1,980 tool-use cases with a twelve-category failure system. Tool initialization was a major weakness for smaller models, while a 32-billion-parameter Qwen model matched GPT-4.1 in the tested procedure. A 14-billion-parameter model reached 96.6 percent success with 7.3-second latency on commodity hardware.
The result argues for diagnostic detail. “The agent failed” is not enough. Teams need to know whether the failure came from tool setup, parameter construction, execution, result interpretation, or a decision not to act.
Recovery changes the result
VLAA-GUI adds three explicit controls: verify completion before stopping, interrupt repeated action cycles, and search for an unfamiliar procedure when needed. The system reached 77.5 percent on OSWorld and 61 percent on WindowsAgentArena. Its loop control nearly halved wasted steps for models prone to repetition.
This is a strong result, but I would not describe it as proof that current agents can generally operate computers. It shows that verification and recovery components materially improve a capable model. Those components are part of the system being evaluated.
GUI vs. CLI compares 440 matched desktop tasks across eighteen applications. The strongest screen-only agent achieved 59.1 percent, while the strongest original command-based agent achieved 48.2 percent. Adding verifier-guided command skills raised the command route to 69.3 percent.
The comparison identifies different limits. Screen-based agents had difficulty with long grounded interaction. Command-based agents depended on whether the available skills covered the task. Neither interface was universally better.
Professional work remains difficult
Workflow-GYM evaluates long tasks in specialized professional software. Even the strongest tested models achieved only slightly above 30 percent success. Common failures included skipped stages, accumulated errors, changed objectives, and weak understanding of the software.
This is the result I would use when reviewing procurement claims. General desktop benchmarks and short browser tasks do not establish readiness for finance, design, engineering, healthcare, or other professional workflows. The software state is more complex, the work lasts longer, and a plausible final screen may still contain a material error.
My production standard
For a defined workflow, I would require repeated execution from the same state, then repeat it with equivalent wording, delayed tools, changed interface positions, partial responses, and one recoverable mistake. I would capture both final state and critical intermediate actions. I would measure unnecessary actions, recovery time, incorrect completion claims, and the exact point where state diverged.
I would also separate model performance from the surrounding system. A verifier, recovery rule, search function, or command skill can improve the result. That improvement is valid, but it belongs in the description of what was tested.
A computer-use agent is ready for a task when it completes that task consistently, proves the required state, and handles ordinary faults without causing an incorrect change. One successful run establishes possibility. Repeated controlled runs establish reliability.
