Introduction

Turn your agent’s failures into tested improvements. Give Beaker your agent and some examples. Beaker runs the agent, finds where it fails, tries changes, and keeps the ones that score higher. You get the winning changes as a pull request along with explanations of what insights it gathered from the experiments.

Use it to fix a production agent you already have, or to find the right model for your task given your cost, latency and model provider requirements. Beaker automatically finds improvements across your harness, memory, tools, skills, …

How does it work?

1. You bring three things: your production agent’s code, datasets of train and test samples, and a way to score an agent prediction.
2. Beaker finds failures. It runs your agent on a set of train examples and looks at what went wrong.
3. Beaker tries hypotheses. Each hypothesis is a change to your agent’s harness, prompt, memory, tools, skills, etc. Beaker tests it and keeps it only if the score goes up.
4. Beaker checks the winner on test examples it never saw, so the final score is honest.
5. You review a draft pull request with the changes. You’re in control: merge it, or don’t.

Getting started

Getting started with Beaker only takes a few minutes:

1. Create an account at app.withbeaker.ai
2. Connect GitHub and install the Beaker GitHub App
3. Paste the prompt from Add an agent into your coding agent

…and your first experiment starts right away. Check out: