Learning Rate
Why it matters
If you can tune only one hyperparameter, tune this one. A wrong learning rate is the most common reason a training run fails, and the failure is easy to misread:
- Too high: the loss spikes, oscillates or becomes
NaN. In fine-tuning, the model can forget what it knew and start producing garbage. - Too low: the loss barely moves. A fine-tune "doesn't take", and the model behaves just like the base model.
Because the best value differs by model, method and data, it's the first thing to check when a recipe that worked for someone else doesn't work for you.
Deep Dive
Training a neural network is like walking downhill in fog. The gradient tells you which direction is down from where you stand. The learning rate decides how far you step in that direction before checking again. Each weight is updated by roughly: new weight = old weight − learning rate × gradient.
Take steps that are too large and you overshoot the valley, bouncing from side to side or flying off entirely. Take steps that are too small and you inch along, wasting compute, and may settle into a poor spot. The right value is somewhere in between, and it's usually a small number like 0.0001 (written 1e-4).
Typical Values
These are starting points with the AdamW optimizer, not rules:
- Pre-training an LLM from scratch: peak around 1e-4 to 6e-4.
- Full fine-tuning of a pre-trained model: 1e-5 to 5e-5. The weights are already good, so move gently.
- LoRA and QLoRA: 1e-4 to 2e-4. The adapter weights start near zero and need larger steps.
Plain SGD uses much larger values, such as 0.1, because it doesn't rescale gradients the way Adam does.
Warmup
Most recipes don't start at the full learning rate. They ramp up from near zero over the first few hundred or thousand steps. Early in training, gradients are large and erratic, and adaptive optimizers such as Adam haven't yet gathered reliable statistics. Full-size steps at that point can knock the model into a bad region it never recovers from. Warmup is cheap insurance, typically 1% to 10% of total steps.
Decay
After warmup, the rate is usually lowered over time, most commonly along a cosine curve or a straight line. Large steps make fast progress early; small steps allow fine adjustments late. The wiki's Learning Rate Schedule entry covers the options.
Interactions
- Batch size: larger batches give less noisy gradients and tolerate higher learning rates. A common rule of thumb is to scale the rate up when you scale the batch up.
- Optimizer: Adam-family optimizers adapt the step for each weight, which makes them more forgiving, but the base rate still matters.
- Weight decay: a separate setting that shrinks weights each step. In AdamW its effect is tied to the learning rate.
- Per-layer rates: some fine-tuning recipes use smaller rates for early layers, which hold general features, than for later ones.
Finding a Good Value
Try a few values spaced by factors of about 3 (for example 1e-5, 3e-5, 1e-4, 3e-4) on short runs and compare the loss curves. A good rate gives a loss that falls quickly and smoothly. The largest value that still trains stably is often close to the best.