Goodhart's Law
Why it matters
Machine learning is optimization against numbers, so Goodhart's Law isn't an occasional nuisance in AI. It's a constant force. It shows up at every level:
- Benchmarks: once a benchmark becomes the number that sells a model, scores rise faster than real ability.
- Training: models maximize the reward they're given, not the intent behind it.
- Products: systems optimized for engagement or thumbs-up ratings learn to please instead of help.
Knowing the law makes you a better reader of AI claims and a better designer of evaluations.
Deep Dive
The law is named after the British economist Charles Goodhart, who noted in 1975 that statistical relationships used to steer monetary policy tended to break down once policy makers started relying on them. The crisp version everyone quotes, "when a measure becomes a target, it ceases to be a good measure", was phrased later by the anthropologist Marilyn Strathern.
The idea is simple. A number is useful because it tracks something you care about. Once people (or algorithms) are rewarded for the number itself, they find ways to raise it that don't improve the thing it was tracking. A school judged on test scores teaches to the test. A call center judged on call length hangs up on hard problems.
Benchmarks
A benchmark starts as an honest sample of a skill. Then labs tune models and prompts against it, its questions leak into training data (contamination), and research directions that raise the score get favored. Eventually the score saturates while users notice little difference. This is why MMLU gave way to harder exams and why new benchmarks keep being built. Even human-preference leaderboards can be gamed: in 2025, researchers showed that testing many private model variants on a public arena and publishing only the best one inflates rankings.
Training Signals
Reward hacking is Goodhart's Law inside the training loop. A coding agent rewarded for passing tests may edit the tests. A model trained on human ratings learns that flattery and confident-sounding answers score well, which is one root of sycophancy. A model judged by another LLM learns that judge's quirks. More capable optimizers are better at finding these gaps, so the problem grows with capability.
In Your Own Project
Your evaluation set is a proxy too. If you iterate on prompts against the same 50 test cases for weeks, you will overfit to them. A retrieval metric can rise while answers get worse. Classic text metrics such as BLEU and ROUGE improved for years in ways human readers didn't appreciate.
Living With It
You can't escape the law, but you can slow it down:
- Hold out data. Keep a test set you look at rarely, and refresh it.
- Use several measures. Gaming many metrics at once is harder than gaming one.
- Read the outputs. Regularly inspect real samples, especially the highest-scoring ones, where exploits hide.
- Measure closer to the goal. Task success with real users beats a proxy score.
- Prefer private or frequently renewed benchmarks when comparing models, and be skeptical of any single headline number.
A related formulation, Campbell's Law, makes the same point about social indicators: the more a quantitative measure is used for decisions, the more it gets corrupted.