Self-Play
Why it matters
Self-play produced some of AI's most famous results. AlphaGo used it to go beyond the human games it started from. AlphaGo Zero (2017) dropped human data entirely: starting from random play and knowing only the rules, it surpassed the version that had beaten world champion Lee Sedol within days, winning 100 games to 0. AlphaZero used the same recipe to master chess and shogi.
It's the clearest demonstration that an AI system can exceed human knowledge instead of imitating it. That matters today because language models learn mostly from human-written text, and researchers are looking for ways to move past that ceiling.
Deep Dive
To get good at a game, you need opponents at your level. A beginner learns nothing from being crushed by a grandmaster, and a grandmaster learns nothing from beating a beginner. Self-play gives an AI the perfect training partner: itself. The agent plays against its current version or earlier versions, learns from the wins and losses, and becomes a slightly better player, which in turn makes it a slightly tougher opponent.
Because the opponent improves in step with the learner, the difficulty rises automatically, and the supply of training games is limited only by compute.
A Short History
- 1959: Arthur Samuel's checkers program at IBM improves by playing itself.
- 1992: TD-Gammon reaches expert-level backgammon through self-play and discovers strategies that human players later adopted.
- 2016–2017: AlphaGo, AlphaGo Zero and AlphaZero.
- 2019: OpenAI Five defeats the world champions at Dota 2 after the equivalent of tens of thousands of years of self-play. DeepMind's AlphaStar reaches grandmaster level at StarCraft II, and the poker AI Pluribus beats professionals at six-player no-limit Texas hold'em.
Why It Works
Self-play creates an automatic curriculum: tasks get harder exactly as fast as the learner can handle. It also needs no labeled data, only a set of rules and a way to tell who won.
How It Goes Wrong
- Chasing its own tail: an agent can learn strategy B to beat A, then C to beat B, then return to A, like rock-paper-scissors, without really improving. The fix is to play against a pool of past versions. AlphaStar used a whole "league" of agents, including some whose only job was to exploit the main agents' weaknesses.
- Blind spots: an agent that has only ever faced itself can be fragile against unfamiliar styles. In 2023, amateur players beat a top Go program using an unusual strategy found by an adversarial AI.
- Needs a clear score: it works where there's a simulator and an unambiguous winner. Most real-world tasks have neither.
Self-Play and LLMs
Applying the idea to language is an active research area. Examples include models that debate each other in front of a judge, attacker and defender models for automated red teaming, negotiation and social deduction games, and setups where one model proposes problems that another must solve, with code execution as the referee. The obstacle is the scoring: text rarely has a clear win condition, so these methods work best where answers can be checked automatically, as in math, code and games.