Skip to main content
Zubnet AI › Learn › Wiki › Self-Play
Training

Self-Play

Also known as: Self-Play Reinforcement Learning
A training method where an AI improves by competing against copies of itself, generating its own ever-harder training data.

Why it matters

Self-play produced some of AI's most famous results. AlphaGo used it to go beyond the human games it started from. AlphaGo Zero (2017) dropped human data entirely: starting from random play and knowing only the rules, it surpassed the version that had beaten world champion Lee Sedol within days, winning 100 games to 0. AlphaZero used the same recipe to master chess and shogi.

It's the clearest demonstration that an AI system can exceed human knowledge instead of imitating it. That matters today because language models learn mostly from human-written text, and researchers are looking for ways to move past that ceiling.

Deep Dive

To get good at a game, you need opponents at your level. A beginner learns nothing from being crushed by a grandmaster, and a grandmaster learns nothing from beating a beginner. Self-play gives an AI the perfect training partner: itself. The agent plays against its current version or earlier versions, learns from the wins and losses, and becomes a slightly better player, which in turn makes it a slightly tougher opponent.

Because the opponent improves in step with the learner, the difficulty rises automatically, and the supply of training games is limited only by compute.

A Short History

  • 1959: Arthur Samuel's checkers program at IBM improves by playing itself.
  • 1992: TD-Gammon reaches expert-level backgammon through self-play and discovers strategies that human players later adopted.
  • 2016–2017: AlphaGo, AlphaGo Zero and AlphaZero.
  • 2019: OpenAI Five defeats the world champions at Dota 2 after the equivalent of tens of thousands of years of self-play. DeepMind's AlphaStar reaches grandmaster level at StarCraft II, and the poker AI Pluribus beats professionals at six-player no-limit Texas hold'em.

Why It Works

Self-play creates an automatic curriculum: tasks get harder exactly as fast as the learner can handle. It also needs no labeled data, only a set of rules and a way to tell who won.

How It Goes Wrong

  • Chasing its own tail: an agent can learn strategy B to beat A, then C to beat B, then return to A, like rock-paper-scissors, without really improving. The fix is to play against a pool of past versions. AlphaStar used a whole "league" of agents, including some whose only job was to exploit the main agents' weaknesses.
  • Blind spots: an agent that has only ever faced itself can be fragile against unfamiliar styles. In 2023, amateur players beat a top Go program using an unusual strategy found by an adversarial AI.
  • Needs a clear score: it works where there's a simulator and an unambiguous winner. Most real-world tasks have neither.

Self-Play and LLMs

Applying the idea to language is an active research area. Examples include models that debate each other in front of a judge, attacker and defender models for automated red teaming, negotiation and social deduction games, and setups where one model proposes problems that another must solve, with code execution as the referee. The obstacle is the scoring: text rarely has a clear win condition, so these methods work best where answers can be checked automatically, as in math, code and games.

← All Terms
ESC