Bias-Variance Tradeoff
Why it matters
The bias-variance tradeoff is the mental model behind most practical training decisions: how big a model to use, how much regularization to add, how long to train and whether more data will help. If your model has high bias, more data won't fix it, but more capacity will. If it has high variance, more data, regularization or simpler models will help.
It also explains why ensembles work. Averaging many high-variance models (as random forests do) cancels out much of their individual noise while keeping their low bias.
Deep Dive
Every prediction error can be split into three parts. Bias is error from wrong assumptions: a model that's too rigid to represent the real pattern, like fitting a straight line to a curve. Variance is error from being too sensitive to the particular training data: train the model on a slightly different sample and you get very different predictions. Irreducible noise is randomness in the data that no model can remove.
The tradeoff is that, in classic machine learning, reducing one tends to increase the other. Simple models have high bias and low variance (they underfit). Complex models have low bias and high variance (they overfit). The goal is the sweet spot in between.
Note that "bias" here is a statistical term. It's unrelated to social or ethical bias in AI, which the wiki covers separately under Bias.
A Dartboard Picture
Imagine throwing darts. Low bias, low variance: tight cluster on the bullseye. High bias, low variance: tight cluster, but off to one side, so you're consistently wrong. Low bias, high variance: centered on average, but scattered everywhere. High bias, high variance: scattered and off-center. Training a model is like adjusting your throw, and each choice moves you between these patterns.
Diagnosing in Practice
- High bias (underfitting): training error is high, and validation error is about the same.
- High variance (overfitting): training error is low, but validation error is much higher.
Cross-validation helps here: if scores swing a lot across folds, variance is a problem.
Deep Learning Bends the Rule
Modern neural networks complicated the classic story. Researchers found double descent: as you keep growing a model past the point where it can memorize the training data, test error first gets worse, then improves again. Very large, heavily overparameterized networks often generalize well despite having enough capacity to memorize everything. Part of the explanation is implicit regularization: training with gradient descent tends to find simple solutions among the many that fit the data.
This is why scaling laws for LLMs look so different from the textbook U-curve. For large models trained on huge, rarely repeated datasets, bigger is usually better, and the main risk is undertraining (bias) rather than overfitting.
Where the Classic Tradeoff Still Bites
The textbook tradeoff is very much alive when data is small: tabular problems, fine-tuning on a few hundred examples, or niche classifiers. In those settings, a smaller model, strong regularization or a simple baseline often beats a big network. Always compare against one before reaching for something larger.