Before You Fix the Agent, Check the Eval
What if the failure you’re trying to fix isn’t actually a failure at all? Evaluations decide what gets fixed, which changes survive, and what ships. When they penalize correct behavior, changing the agent to pass can teach it to make a mistake.
Across two runs on 120 training examples from the revised AutomationBench dataset, Beaker improved the training score by about 14 percentage points per run. It also flagged eight evaluation concerns for human review, without being told which examples were faulty. On LiveNewsBench, Beaker flagged 20 of 300 training questions across five runs, including six confirmed grader errors.
Beaker investigates failures while proposing and testing improvements to your agent. When it suspects an evaluation problem, it gives you evidence and a suggested correction.
The grader counted all 12 events instead of the four new bookings.
In one task from the 120-example experiment, the agent was given a calendar with eight existing meetings and asked to add four dock bookings. It correctly left 12 events, but the grader (the code that checks the agent’s work) expected just four events in the entire calendar.
Figure 1. The calendar is correct. The proposed correction changes which events the grader counts.
Task instructions · Incorrect count check · Grader code
Changing the agent to pass this check could teach it to delete the eight existing meetings. Beaker identified the mismatch during optimization and flagged it for human review.
Across these experiments and our work with production agents, we found four ways a test can reject correct work:
- Brittle wording checks. A correct date or count fails because the grader expects different wording.
- Wrong or stale expected answers. In our order-review case study, a reviewer removed two valid items, making the expected answer incorrect.
- Conflicting requirements. The task permits an action that the grader rejects.
- Inconsistent grading. The same answer receives different verdicts across runs.
In the AutomationBench pilot, Beaker flagged all six cases that penalized correct work.
Production evaluation sets often start with a handful of examples, so a few faulty checks can steer optimization toward the wrong behavior. We used a small AutomationBench pilot to test whether Beaker could flag checks that penalized correct work while leaving valid controls unflagged.
In an 11-example pilot, Beaker flagged all six cases that penalized correct work and left all four valid controls unflagged, without being told which examples were faulty. It missed one defect that rewarded incorrect work.
Excluding the six flagged examples raised the score from 74.4% to 100% on the remaining five, using the same agent outputs. This is a rescore, not an agent improvement, and still includes the missed defect.
The pilot combined three original AutomationBench examples with eight from the revised dataset. View public pilot run.
Beaker improved training scores and flagged eight concerns on 120 examples.
In the larger AutomationBench experiment, we ran Beaker twice on the same 120 training examples from Prime Intellect’s revised AutomationBench dataset. Each run improved the training score by about 14 percentage points. These are training scores, not held-out results.
The eight evaluation concerns included five confirmed grading or task-data defects and three unclear requirements for human review. A separate integration bug is not included in that count.
Both runs started from the same agent code, but the second carried forward six dataset flags and added two: the confirmed calendar-count defect and a future-price disclosure concern that still needs a human decision.
View first 120-example run · View second 120-example run
LLM judges can be brittle and inconsistent.
Beaker also found evaluation problems when an LLM graded the answers. LiveNewsBench asks agents to research recent news. GPT-4.1 acts as the judge, comparing their answers with reference answers using the SimpleQA grading prompt.
One question asked which country’s prime minister tied recognition of Palestine to elections in 2026 excluding Hamas. An agent answered “Canada (Prime Minister Mark Carney).” The reference answer was “Canada,” yet the grader rejected this answer and close variants across all four agent stacks we tested.
Figure 2. Beaker flagged a matching answer rejected by the LLM judge. The question is summarized; the comparison illustrates the reported finding.
Across five optimization runs on 300 training questions, Beaker flagged 20 distinct questions for review, using the original dataset and grader unchanged. It was not told which questions were faulty. Our review confirmed six grader errors: three rejected matching or equivalent answers, and three gave the same answer different verdicts across runs.
Eleven additional findings raised concerns about potentially stale reference answers or questions with more than one valid interpretation. Beaker supplied evidence and suggested corrections for human review. These concerns still need checks against the original sources. We set aside three other flags as weak.
Optimizing against the Canada verdict could teach the agent to strip correct context from its answer instead of fixing the grader. Beaker flags that mismatch for human review.