Skip to main content
Zubnet AI › Learn › Wiki › Vanishing Gradients
Fundamentals

Vanishing Gradients

Also known as: Exploding Gradients, Vanishing Gradient Problem
A training failure where the learning signal shrinks to nothing (or blows up) as it travels backward through many layers.

Why it matters

Much of modern architecture exists to solve this one problem. ReLU activations, careful weight initialization, normalization layers, residual connections, LSTMs and gradient clipping were all, at least partly, answers to it. Understanding vanishing gradients explains why a transformer block looks the way it does.

It's also still a practical issue. When a training run stalls with a flat loss, or dies with NaN after a few hundred steps, gradient problems are among the first suspects.

Deep Dive

Neural networks learn through backpropagation: the error at the output is passed backward, layer by layer, telling each weight how to change. At every layer the signal gets multiplied by that layer's local derivatives. Multiply many numbers smaller than 1 and the result heads toward zero; the early layers get almost no signal and stop learning. That's the vanishing gradient problem. Multiply many numbers larger than 1 and the signal grows without limit, weights jump wildly and the loss turns into NaN. That's the exploding gradient problem.

Both get worse with depth, and for years they were the main reason deep networks didn't work.

Where It Comes From

The problem was identified in the early 1990s by Sepp Hochreiter and others, mainly in recurrent networks. An RNN applies the same weights at every time step, so backpropagating through 100 steps is like passing through 100 layers. Older activation functions made it worse: the derivative of a sigmoid is at most 0.25, so each layer shrinks the signal by at least a factor of four.

The Fixes

  • Better activations: ReLU has a derivative of exactly 1 for positive inputs, so it doesn't shrink the signal. GELU and SwiGLU, used in modern LLMs, behave similarly.
  • Initialization: Xavier and He initialization choose starting weights so signals keep roughly the same scale across layers.
  • Normalization: batch norm and layer norm re-center and rescale activations, keeping them in a healthy range.
  • Residual connections: adding a layer's input to its output gives gradients a direct path backward. This is what made 100+ layer networks, and every transformer, trainable.
  • Gated architectures: LSTMs and GRUs let information flow through time with little decay.
  • Gradient clipping: if the gradient's size exceeds a threshold, scale it down. The standard fix for explosions, used in nearly every LLM training recipe.

In Today's Models

Transformers largely tamed the problem by combining residual connections, layer norm and careful initialization. Small choices still matter: putting normalization before each sub-layer ("pre-norm") instead of after made very deep transformers much more stable. Learning rate warmup also exists partly to avoid exploding updates in the first steps.

Spotting It

Log the gradient norm during training. A norm that collapses toward zero suggests vanishing; sudden spikes followed by NaN suggest exploding. Usual first responses are lowering the learning rate, adding or tightening gradient clipping, lengthening warmup and checking mixed-precision settings.

← All Terms
ESC