From the lab notebook
Notes on turning agent failures into experiments, and experiments into fixes.
- Research
Before You Fix the Agent, Check the Eval
Across two runs on 120 training examples from AutomationBench, Beaker flagged eight concerns for human review and improved the agent’s training score by about 14 percentage points per run.
Ella Hofmann-Coyle - Research
Speeding up agent execution with Beaker
Beaker made agent execution 36.6% faster and lowered model cost per scenario by 34.6%, while keeping the model fixed and maintaining a similar score on the test tasks.
Thomas Vujasinovic - Research
Adventures in Weight-Space Programming with Beaker
Can a model fix another model by editing its weights? We ran the weight-space programming experiment in Beaker. Without ever running the model, Beaker found a one-neuron fix for Qwen3.5-0.8B's math loops, and we took the edits apart to see what they really change.
Reza Nayebi - Case Study
Building a Business-Critical Agent in One Day with Beaker
We used Beaker to build an agent that proofreads our order extraction agent's work. Beaker optimized it to 95% on a 93-case eval and we deployed it to production in under a day. In its first day live, it stopped two bad orders from reaching customers.
Julian George - Research
Don't wait for the next model
We tuned our agent's harness to work well with a model from a few generations ago, and it beat the latest and much more capable model. The capability you are waiting for may already be in today's models.
The Beaker Team - Perspective
The case against training your own models (for now)
Data leakage to model providers is a real concern, but it's a procurement problem, not a reason to train your own weights. Improving an AI system is a resource allocation question, and the answer is an experiment, not an opinion.
Victor Sanh - Research
Tune the agent before you judge the model
An agent tuned for one model can make other models look worse than they are. Beaker autonomously tuned an AppWorld agent separately for a variety of models: the gap between them went from 61 points to 18, the ranking changed, and the choice came down to cost and speed.
The Beaker Team