The intuition: random training order means the model encounters hard examples before it has the foundation to learn from them, wasting gradient updates. By starting with easy, clear examples, the model builds basic representations that make harder examples learnable. Bengio et al. (2009) formalized this, showing that curriculum training converges faster and sometimes reaches better solutions than random ordering.
Modern LLM training uses a form of curriculum learning through data mixing schedules. Early training might use a broad mix of web text. Later stages increase the proportion of high-quality data (curated text, code, math, reasoning). Some training recipes "anneal" on premium data in the final phase — the model learns broad patterns first, then refines on carefully selected examples. This data scheduling is one of the most closely guarded secrets of frontier labs.
Self-paced learning is a variant where the model itself decides what's easy or hard, based on its current loss on each example. Examples with low loss (already learned) are deprioritized, while examples with moderate loss (challenging but learnable) are emphasized. Examples with very high loss (noise, mislabeled data) are also deprioritized as potential outliers. This adaptive curriculum adjusts automatically as the model improves.