Skip to main content
Zubnet AI › Learn › Wiki › Gradient Accumulation
Training

Gradient Accumulation

Also known as: Accumulated Gradients, Virtual Batch Size
A training trick that simulates a large batch on limited memory by adding up the gradients of several small batches before updating the weights.

Why it matters

This is one of the everyday tools that make fine-tuning possible on consumer hardware. It appears in almost every training config as gradient_accumulation_steps. The quantity that matters for training behavior is the effective batch size:

effective batch size = per-step batch size × accumulation steps × number of GPUs

When a guide says "batch size 32", it means the effective one. Knowing this lets you reproduce a recipe designed for a big GPU on a small one.

Deep Dive

Training normally works in a loop: take a batch of examples, compute the gradients, update the weights, repeat. Bigger batches give smoother, more reliable gradients, but every example in a batch takes GPU memory. If your GPU fits only 4 examples and the recipe calls for 32, you have a problem.

Gradient accumulation solves it. Process a batch of 4, compute its gradients, but don't update yet. Process another 4 and add the new gradients to the old. After 8 such mini-batches, the summed gradients are the same as one batch of 32 would have produced. Now update the weights and reset. Memory use stays at the level of 4 examples.

How It Looks in Code

In PyTorch, gradients add up by default until you clear them, so accumulation is just a matter of when you call the optimizer:

for i, batch in enumerate(loader):
    loss = model(batch).loss / accum_steps   # scale so the sum is an average
    loss.backward()                          # gradients accumulate
    if (i + 1) % accum_steps == 0:
        optimizer.step()
        optimizer.zero_grad()

Libraries such as Hugging Face Transformers do this for you when you set gradient_accumulation_steps.

What It Costs

Accumulation saves memory, not time. The 32 examples still pass through the model, just 4 at a time. On hardware where a real batch of 32 would fit, the real batch is faster because it uses the GPU's parallelism better.

Details That Trip People Up

  • Steps mean updates. With accumulation, one optimizer step covers several mini-batches. Learning rate schedules, warmup and logging usually count optimizer steps, so an epoch has fewer of them.
  • Batch normalization. Layers that compute statistics over the batch still see only the small batch. Transformers use layer normalization, which is unaffected.
  • Contrastive learning. Methods that compare every example in a batch with every other one, as in CLIP or embedding training, gain nothing from accumulation, because the examples in different mini-batches never meet.
  • Leftover batches. If the dataset size isn't a multiple of the accumulation count, the final partial group needs handling.

Choosing Values

Set the per-step batch size as high as memory allows, then use accumulation to reach your target effective batch size. For LoRA fine-tuning of LLMs, effective batch sizes of 16 to 64 are common. Accumulation combines well with the other memory savers: gradient checkpointing, mixed precision and quantized base models as in QLoRA.

← All Terms
ESC