Small Models Can Run Agents with Search, Tools, and Cloud Support

Scope note: This review covers seven recent studies of small models used as agents on devices, in specialist roles, and with selective cloud support. It does not claim that a small model can replace a frontier model for every task.

The most useful agent may not be the largest model available.

Seven recent preprints test models below ten billion parameters in tool use, search, terminal work, on-device response, and mixed local-cloud systems. The strongest results do not come from asking a small model to imitate a frontier model across every task. They come from changing the system around the model.

The design uses narrow roles, consistent tool behavior, targeted training, and escalation when the local model lacks enough capability.

Small agents need explicit tool habits

Search, Do Not Guess finds that small models search less often than larger models even though they contain less factual knowledge. That combination increases unsupported answers. The authors train small models to retrieve consistently and answer from the evidence. The method improved results by 17.3 points on Bamboogle and 15.3 points on HotpotQA, reaching results comparable to larger models in the tested tasks.

An adaptive policy performed worse in this setting. The small model was not reliable enough to decide when search was unnecessary. A consistent requirement to retrieve produced better results.

TinyLLM evaluates tool and API calls across small model sizes and several training methods. Models between one and three billion parameters substantially outperformed those below one billion. Hybrid training reached 65.74 percent overall and 55.62 percent on multi-turn work.

The result defines a practical lower range for the tested behavior. Extreme compression can remove too much capacity for stable multi-step tool use.

One small agent can be better than a small group

Rethinking Scale compares models below ten billion parameters in three configurations: model alone, one tool-equipped agent, and several collaborating agents. The single tool-equipped agent provided the best balance of result and cost. Multiple agents added communication and execution expense with limited improvement.

More agent roles do not guarantee more capability. A multi-agent design is justified when specialization or parallel work provides a measured gain.

Youtu-LLM takes a training-first approach. The 1.96-billion-parameter model receives long-context support and staged training in general language, technical reasoning, planning, and tool use. The paper reports leading results among sub-two-billion-parameter models and stronger agent-specific performance than comparable small models.

This result shows that agent behavior can be trained directly at small scale. It does not show that parameter count has stopped mattering. The training data and curriculum contribute to the capability.

Specialist subagents can reduce frontier-model work

Terminus-4B trains a four-billion-parameter model for terminal execution inside a larger coding-agent system. The specialist handles search, build output, tests, and other verbose command work. It reduced the main agent’s token use by about 30 percent without reducing performance on the reported software benchmarks. In some configurations it matched or exceeded frontier models assigned to the same subtask.

This is a strong use of a small model because the role is bounded and its output can be checked. The main agent retains the broader requirement and decides what to do with the result.

Local and cloud models can divide one response

Micro Language Models Enable Instant Responses trains models from eight to thirty million parameters to generate the first four to eight words of a response on a constrained device. A cloud model then continues the sentence. The system includes recovery methods for cases where the local opening is unsuitable.

The local model does not solve the full task. It reduces perceived delay while the stronger model works. This is useful for watches, glasses, and other devices that cannot continuously run a conventional language model.

When Cloud Agents Meet Device Agents studies mixed systems across accuracy, cloud cost, and device energy. Small models benefited from cloud help, but the best division varied by task. Additional frontier computation did not consistently improve performance.

That final result is important. Escalation needs a reason and a measurable benefit. Sending every decision to the cloud removes much of the privacy, latency, and cost value of local execution.

My design rule

I would assign a small model a task when the action space is limited, evidence can be retrieved, output can be verified, or escalation is available. I would test multi-turn consistency, tool-call arguments, unsupported answers, energy use, latency, and the frequency of cloud requests.

I would not evaluate the model in isolation if the production system includes search, tools, memory, or a larger supervisor. Those components are part of the result. I would also compare one specialist with several small agents because coordination costs can exceed the gain.

Small models can provide private, fast, and inexpensive agent behavior. The current evidence supports specific system designs, not a general replacement claim. A small model becomes useful when its responsibility is exact and the system handles the work it cannot perform reliably.