In practice, stochastic gradient descent processes the training data in random mini-batches. Each batch gives an estimate of the true gradient — larger batches give better estimates (less noise) but cost more memory and compute per step. Typical batch sizes range from 32 (small models, single GPU) to millions of tokens (LLM pre-training across thousands of GPUs).
LLM pre-training uses enormous effective batch sizes (millions of tokens per update) distributed across many GPUs. At this scale, the learning rate must be carefully tuned — the linear scaling rule (double the batch size, double the learning rate) works up to a point, then breaks down. Gradient accumulation lets you simulate large batches on smaller hardware by accumulating gradients across multiple forward passes before updating.
Modern LLM pre-training typically runs for less than one epoch on the full dataset — the data is so large that the model never sees all of it. This is a shift from classical ML where 10–100 epochs was normal. Research suggests that repeating data (multiple epochs) can actually hurt LLM performance due to memorization effects, though this depends on data quality. Fine-tuning, by contrast, typically runs for 1–5 epochs on a much smaller dataset.