Gradient Accumulation
Why it matters
This is one of the everyday tools that make fine-tuning possible on consumer hardware. It appears in almost every training config as gradient_accumulation_steps. The quantity that matters for training behavior is the effective batch size:
effective batch size = per-step batch size × accumulation steps × number of GPUs
When a guide says "batch size 32", it means the effective one. Knowing this lets you reproduce a recipe designed for a big GPU on a small one.
Deep Dive
Training normally works in a loop: take a batch of examples, compute the gradients, update the weights, repeat. Bigger batches give smoother, more reliable gradients, but every example in a batch takes GPU memory. If your GPU fits only 4 examples and the recipe calls for 32, you have a problem.
Gradient accumulation solves it. Process a batch of 4, compute its gradients, but don't update yet. Process another 4 and add the new gradients to the old. After 8 such mini-batches, the summed gradients are the same as one batch of 32 would have produced. Now update the weights and reset. Memory use stays at the level of 4 examples.
How It Looks in Code
In PyTorch, gradients add up by default until you clear them, so accumulation is just a matter of when you call the optimizer:
for i, batch in enumerate(loader):
loss = model(batch).loss / accum_steps # scale so the sum is an average
loss.backward() # gradients accumulate
if (i + 1) % accum_steps == 0:
optimizer.step()
optimizer.zero_grad()
Libraries such as Hugging Face Transformers do this for you when you set gradient_accumulation_steps.
What It Costs
Accumulation saves memory, not time. The 32 examples still pass through the model, just 4 at a time. On hardware where a real batch of 32 would fit, the real batch is faster because it uses the GPU's parallelism better.
Details That Trip People Up
- Steps mean updates. With accumulation, one optimizer step covers several mini-batches. Learning rate schedules, warmup and logging usually count optimizer steps, so an epoch has fewer of them.
- Batch normalization. Layers that compute statistics over the batch still see only the small batch. Transformers use layer normalization, which is unaffected.
- Contrastive learning. Methods that compare every example in a batch with every other one, as in CLIP or embedding training, gain nothing from accumulation, because the examples in different mini-batches never meet.
- Leftover batches. If the dataset size isn't a multiple of the accumulation count, the final partial group needs handling.
Choosing Values
Set the per-step batch size as high as memory allows, then use accumulation to reach your target effective batch size. For LoRA fine-tuning of LLMs, effective batch sizes of 16 to 64 are common. Accumulation combines well with the other memory savers: gradient checkpointing, mixed precision and quantized base models as in QLoRA.