Regularization
Why it matters
Regularization is the main lever against overfitting. Without it, deep networks trained on modest datasets would often learn their training data perfectly and fail on everything else. Nearly every training recipe uses some form of it, and when you read a fine-tuning config, settings like weight_decay, dropout and lora_dropout are all regularization.
It's also a dial, not a switch. Too little and the model overfits; too much and it underfits. Tuning regularization strength is one of the most common hyperparameter jobs in practice.
Deep Dive
A model with enough capacity can memorize its training set, noise and quirks included. Regularization is the umbrella term for everything we do to stop that and push the model toward simpler, more general solutions. Some techniques add a penalty to the loss, some inject noise, some limit training, and some simply give the model more varied data.
The idea traces back to a principle older than machine learning: when two explanations fit the facts, prefer the simpler one.
Penalty-Based: L2 and L1
L2 regularization adds a penalty proportional to the squared size of the weights, which pushes them toward small values. Large weights let a model react sharply to tiny input changes, which is a hallmark of memorization. L1 regularization penalizes the absolute size, which tends to push many weights to exactly zero, giving sparse models.
Weight decay is the form you'll see in modern deep learning: each step, weights shrink slightly toward zero. With plain SGD it's equivalent to L2, but with Adam it isn't, which is why AdamW applies weight decay separately from the gradient update. AdamW with a weight decay around 0.01 to 0.1 is the default for most transformer training.
Noise-Based: Dropout and Friends
Dropout randomly turns off a fraction of neurons during training, so the network can't rely on any single path. Related tricks include stochastic depth (randomly skipping whole layers) and label smoothing (telling the model the right answer is 90% likely rather than 100%, so it doesn't get overconfident). Large LLM pre-training often uses little or no dropout, because with trillions of tokens the model rarely sees the same example twice and overfitting is less of a concern.
Data-Based and Procedural
Data augmentation creates varied copies of training examples (crops, flips, paraphrases), so the model learns what doesn't matter. Early stopping halts training when validation loss stops improving. Even the choice of a small batch size has a mild regularizing effect, because noisier gradients keep the model from settling into sharp, overly specific solutions.
Regularization in Fine-tuning
Fine-tuning a big model on a small dataset is where regularization matters most for builders. Parameter-efficient methods like LoRA are themselves a form of regularization: by only allowing low-rank changes, they limit how far the model can drift. Keeping the number of epochs low (often 1 to 3), using a modest learning rate and holding out a validation set are the practical defaults.