Continuous Wiki Improvements with Beaker
A wiki is at the core of one of our production agents, informing millions of user queries and LLM calls. This agent helps warehouses by extracting structured order data from email threads.
Maintaining its wiki was painful: we were constantly fighting stale, disorganized, and bloated content, and every manual change risked regressions in the agent’s outputs. Whenever volume spiked or errors emerged, we'd sometimes have to pull an engineer off their work for a day (or several) just to keep the wiki up to date.
Now, we don't wait for something to break. Beaker automatically improves our wiki, week over week.
Beaker starts from our agent's mistakes, traces each one back to the wiki problem that caused it, and fixes it. Now, instead of spending hours maintaining markdown files, our engineers have more time for building features and solving harder problems.
Our agent's mistakes are usually caused by wiki problems
When we investigated our agent's mistakes, we found these kinds of wiki problems:
- Outdated guidance: A rule changes, but the wiki still tells the agent to follow the old one.
- Conflicting instructions: Two pages describe different ways to handle the same situation, leaving the agent to choose between them.
- Context bloat: Duplicated instructions and irrelevant details make it harder for the agent to find and follow the guidance that matters.
- Vague instructions: Poorly worded rules lead the agent to act inconsistently.
- Mis-scoped SOPs: Policies applied either too broadly or too narrowly.
This wiki is for our extraction agent, which helps 3PL warehouses convert emails into orders. Its wiki contains extraction policies tailored to each organization, facility, and customer's specific business rules. However, these business rules often change behind our backs (e.g., a customer starts sending differently-formatted order PDFs). Before Beaker, whenever this happened, we had to catch the issue, debug it, then manually fix the relevant wiki content.
Correcting these errors manually was time-consuming, and it prevented us from scaling, so we decided to set up Beaker to find and fix these errors for us.
Beaker improves your wiki every week
Every weekend, Beaker uses the previous week's mistakes to find and test wiki improvements.
Humans review our agent’s outputs, so we know our agent has made a mistake if a human needs to correct it. At the end of every week, we automatically build a dataset with those corrected orders and feed it to Beaker. Beaker runs the agent with that data, identifies where its outputs are wrong, and iterates until those errors are resolved.
In one example, Beaker caught an overly specific rule that led three orders to have shipping dates that were weeks too early:
Example 1:
EARLIEST_SHIP_DATE: For <address name> orders, use the shipment date in the latest email subject when that subject names <address name> and gives a date, even if the attached packing slips show a different `SHIP DATE`. Otherwise use the document's `SHIP DATE` field. If a later message explicitly reschedules the shipment, use the newest stated date instead.
EARLIEST_SHIP_DATE: Use the shipment or pickup date in the latest email subject when it identifies the shipment, even if the packing slip has a different `SHIP DATE`. This applies to all receivers. A later explicit pickup or reschedule instruction takes precedence; resolve relative dates from that message's date. Otherwise use the document's `SHIP DATE`.
- Order 1Before After
- Order 2Before After
- Order 3Before After
The more specific ship date guidance may have been correct in the past, but Beaker recognized that it had become outdated and updated it accordingly. This was just one of several stale rules corrected during this run.
Before we merged these changes into prod, we wanted to have confidence that they would work. We wanted to know that they wouldn’t cause any regressions. So, we looked at the full Beaker run results:
We saw that fixing the stale guidance increased the average score, with no significant regressions in cost or latency.
Beaker can clean up redundant content
Wikis easily become bloated. Information can be duplicated, unnecessary details can clutter documents, and a knowledge base’s structure can become confusing and disorganized.
For a while, we suspected that our agent’s wiki was bloated. We wanted to streamline it, but it wasn’t worth the time and risk of regressions.
We decided to use Beaker to streamline our wiki, since it can do so quickly and confidently. To do this, we changed the evaluation score to also penalize wiki size, giving Beaker an incentive to remove unnecessary instructions.
In the resulting experiment, Beaker reduced the extraction wiki’s size by 23.8%. It found that instructions for extracting a customer’s emails were duplicated in both the customer’s file and their facility’s file. It deleted those instructions from the facility file and kept the customer one.
This streamlined knowledge base will be easier to maintain going forward, keeping files lightweight and preventing future drift between multiple sources of truth.
Great Beaker runs need great datasets
To discover and diagnose stale wiki data without causing regressions, it’s important to give Beaker the right dataset.
When designing your experiment’s dataset, here are a few things to consider in order to confidently get changes that work in prod:
- How do you decide when to split cases into different datasets?
- Does your dataset cover the relevant prod scenarios?
- What data belongs in the training set vs in the test set?
We split cases into one dataset per organization, since organizations often follow vastly different business rules. The trade-off is that Beaker may treat a platform-wide issue as specific to a single organization.
For coverage, our #1 priority is including every incorrect order so it can be fixed. We put all of that week’s incorrect orders in the dataset, then pad it with enough correct orders to catch regressions.
We keep the training and test sets identical for the extraction agent. Our datasets are small (<100 samples) and case-by-case variance is significant, so testing on the same data we train on helps us catch regressions. The risk is overfitting, but that's acceptable here: business rules change frequently, and adapting quickly matters more than long-term generalization.
Beaker also helps us prune bad data. For example, in our scheduled run, it flagged a grading defect: our shipping-instructions check gave no credit to a complete, correct set of instructions just because they were in a different order. Cases like this can cause Beaker to make misguided changes, so we filter them out.
Step 1/4
Orders from the week
Reviewed orders from the week
It’s important to keep your wiki clean
Research suggests that a messy wiki is not just a time sink: it can degrade your agent's performance. Recent papers point to three ways this can happen:
- Inaccurate or incomplete knowledge. In DeepRefine: Agentic Knowledge Refinement via Reinforcement Learning, Huang et al. describe how incomplete or incorrect knowledge bases degrade retrieval and downstream task performance, and how those defects compound with use.
- Bloated knowledge bases. In EnterpriseRAG-Bench: A RAG Benchmark for Company Internal Knowledge, Sun et al. find that retrieval quality declines as a knowledge base grows, in a simulation that scaled a company's knowledge base from 5,000 to 500,000 documents.
- Stale instructions. In When Instruction Files Go Stale: Documentation Debt and Hidden Cost in Coding Agents, Mehtiyev et al. find that agents follow stale guidance when it's relevant to their task, even when correct information is available elsewhere, leading to extra tool calls and cost.
While minor on a small scale, these inefficiencies can compound into significant, business-impacting issues as wikis and their systems grow.
Beaker not only saves engineering time by turning wiki maintenance into steady, week-over-week improvement, but its methodical wiki optimization can lead to impactful performance improvements for your agents.