The mechanism: a model trained on real data captures the distribution imperfectly — it overestimates some patterns and misses others. When a second model trains on the first model's outputs, it captures the first model's imperfect distribution, amplifying the errors. By generation 5 or 10, the distribution has collapsed to a narrow, distorted version of the original. Shumailov et al. (2023) demonstrated this empirically across multiple model types.
The practical concern: pre-training datasets are typically scraped from the web, and the web increasingly contains AI-generated content. If 20% of a training corpus is AI-generated, and that AI content has the same statistical biases as the model being trained, those biases get reinforced. The result isn't catastrophic failure but gradual homogenization — models that sound more and more like each other and less like the diversity of human expression.
Solutions include: detecting and filtering AI-generated content from training data (hard at scale), mixing AI-generated data with verified human data (maintaining a "human data floor"), watermarking AI outputs to enable filtering, and maintaining curated, AI-free reference datasets. Some researchers argue that model collapse is overstated if data is properly diversified and quality-controlled, but the risk is taken seriously enough that major labs invest in data provenance.