Speeding up agent execution with Beaker

If your agent works but is too slow or too expensive, Beaker can speed it up without changing the model. On AppWorld it made our agent 36.6% faster and cut model cost per scenario by 34.6%, in 2h24 of automated optimization.

The latency experiment

AppWorld is a benchmark where agents complete everyday tasks across simulated apps such as Spotify and Venmo. The AppWorld agent used GPT-6-Luna to write Python code against their APIs, while Beaker optimized its instructions and tools without changing the model.

You can explore the full run, including every hypothesis Beaker tested and the code changes it kept. The goal completion score measures how fully the agent achieves the task and scenario goals. It stayed similar: 87.5% before optimization and 88.0% after. The optimized agent made fewer model calls and processed fewer input tokens. Beaker found changes that helped it do less unnecessary work.

Our earlier post explored how tuning the work around a model can improve an agent’s capabilities. The same idea applies when quality is already acceptable and the next priority is speed, as a proxy for cost.

Unnecessary work adds up

For instance, in one Spotify training scenario, the agent looked up the same login and recommendation API documentation three times, once for each sub-task. After optimization, it fetched each specification once and reused it across the scenario. With the full set of retained optimizations, the recorded trace went from 21 model calls to 8, while all task checks still passed.

Change how the agent works

We kept the model and evaluation tasks fixed. Beaker measured how changes to the agent’s prompts and tools affected execution time, model cost and quality. The automated optimization run took 2 hours 24 minutes, including baseline and held-out evaluation.

During the run, Beaker tested 10 hypotheses and kept four changes to the agent’s prompts and tools.

Reduce unnecessary context

Beaker kept full results in Python variables and returned only the observations needed for the next decision. It also grouped related lookups and reads into fewer steps. This reduced the context sent back to the model and avoided unnecessary model calls.

Remove bloat from instructions

Beaker replaced a six-step walkthrough with three grouped phases for discovery, login and execution. The shorter example showed the agent how to complete related steps together instead of pausing for a new model call after each operation.

For example, the Spotify playlist-count walkthrough changed as follows:

• Before — 6 steps: discover APIs → read login docs → get credentials → log in → fetch playlists → submit the count.
• After — 3 phases: discover APIs and authentication docs → retrieve credentials, log in and read task-specific docs → fetch playlists and submit the count.

Reuse API documentation

Beaker modified the agent’s tools to retain API documentation within a scenario. The agent could then reuse that documentation across tasks instead of repeating the same lookups, while still fetching fresh task data.

Avoid unnecessary correction turns

Beaker clarified which arguments each tool accepts and when authentication is required. That helped the agent avoid rejected tool calls and the extra model turns needed to correct them.

Latency as a useful signal about cost

When a change removes model calls or reduces the amount of information processed, it can improve speed and cost together. Latency gives you a useful signal about wasted work: reducing it usually means processing fewer tokens, which also lowers cost.

The experiment’s lower input-token count and fewer model calls show how latency optimization can reduce model work.

Results

We evaluated the original and optimized agents on 19 test scenarios, running each scenario three times per version. We averaged the results to account for variation between executions.

The optimized agent ran 36.6% faster, with mean execution time falling from 73.59 to 53.88 seconds per scenario.

Fewer tool calls and tokens also brought 34.6% lower model cost per test scenario.

In a separate AppWorld experiment, Exo reported 13% less execution time and lower model cost while maintaining task completion rates, supporting the same principle of improving how an agent uses its model.

Start with your agent and a set of evaluation tasks. Sign up for Beaker for free to find changes that make your agent faster and lower its model cost.