The case against training your own models (for now)

There is a part of the internet working hard to convince you that you need to train your own models for pretty much all enterprise use cases under the sun. The argument is usually along these lines: your (or your customers') institutional/tribal knowledge leaks to the model provider through every prompt and interaction. As such, model providers are capturing the compounding value of your usage. And the prescribed remedy is training your own weights, or at least fine-tuning them.

The leakage concern is real, but training your own model is not necessarily the fix. If the provider can't store your traffic, they can't learn from it unless you opt in. Zero data retention + no-training-on-my-data-beyond-improving-my-own-usage policies solve it. That's a procurement problem, not an ML problem. The same goes for distillation bans.

But leakage is only half of the argument. The other half is that stopping the provider from learning doesn't make you learn. Your end-user corrections, your traces, your evals should compound into improving your AI systems. Where I disagree is the assumption that compounding necessarily means updating model weights.

Improving an AI system is a finite resource allocation problem. The constraints are engineering time, compute budget, and the accuracy targets or SLAs you've committed to. Today you spend those resources mostly in human form: engineering cycles and experts you hire. In the future you spend them as intelligence: tokens and GPU hours running improvement experiments. And then the question is simple: where does the next dollar of intelligence buy the most improvement?

That's an experimental question, and you can't answer it with opinion alone.

The answer heavily depends on where you are in your AI agent journey. Early on, the next dollar is likely best spent outside of the underlying model: capturing tribal knowledge into wikis/prompts, better context building and orchestration, swapping the underlying model including towards oss models — especially if provider lock-in is a concern. These levers are easily reversible, and more importantly, testable.

Training usually makes sense at a later maturity stage. Your agent architecture is stable and you've hit one of these walls:

• capability is out of distribution for the current generation of models
• behavior can't be reliably specified in natural language or in-context
• usage volume is high enough that re-paying for the same context on every call stops being economical

All three are falsifiable claims. You run the experiment at small scale, you get a number, the number decides. The alternative is overspending on belief in the wrong lever. And in the future, running these experiments will itself be a matter of spending tokens and some compute.