Skip to main content
Zubnet AI › Learn › Wiki › Underfitting
Training

Underfitting

Also known as: High Bias
When a model is too simple or too undertrained to capture the patterns in its data, so it does badly even on the examples it trained on.

Why it matters

Telling underfitting apart from overfitting is one of the most useful diagnostic skills in machine learning, because the fixes point in opposite directions. Overfitting calls for less capacity, more regularization or more data. Underfitting calls for more capacity, longer training or less regularization. Apply the wrong fix and you make things worse.

In the LLM era, underfitting is less about tiny models and more about undertraining. The Chinchilla research showed that many large models of the time had been trained on far too few tokens for their size. They were underfit, and smaller models trained on more data beat them. Today, small models are often deliberately trained far past the "compute-optimal" point precisely to squeeze out every last bit of underfitting.

Deep Dive

Overfitting gets most of the attention, but its opposite is just as common. An underfit model hasn't learned enough. Its training loss stays high, its validation loss stays high, and the two are close together. It's like a student who skimmed the textbook: they don't ace the practice questions or the exam, because they never really got the material.

The classic picture is fitting a straight line to data that clearly curves. No amount of extra data helps, because a straight line simply can't express the curve.

How to Spot It

Look at your loss curves:

  • Underfitting: training and validation loss are both high and close together, and may still be falling when you stop.
  • Overfitting: training loss keeps dropping while validation loss flattens or rises, so the gap widens.
  • Good fit: both are low and the gap is small.

If training loss itself is poor, the model can't even fit the data it has seen. That's underfitting, and adding more data won't fix it.

Common Causes

  • Too little capacity: too few parameters or layers for the complexity of the task.
  • Too little training: stopping too early, or a learning rate so small the model barely moves.
  • Too much regularization: heavy dropout or weight decay can hold a model back from learning real patterns.
  • Weak features or bad inputs: if the information needed to solve the task isn't in the input, no model can learn it.
  • A bug: shuffled labels, a broken data pipeline or a frozen layer often look exactly like underfitting. Check this first.

Fixes

Train longer, raise the learning rate (carefully), use a bigger model, reduce regularization, or improve the inputs. In fine-tuning, a LoRA rank that's too small or too few steps can underfit: the model barely changes its behavior. Try overfitting a tiny subset of your data on purpose first. If the model can't even memorize 100 examples, something in your setup is broken.

Underfitting in Fine-tuning

When people say a fine-tune "didn't take", the model is usually underfit to the new task. Signs include outputs that ignore your target format or still sound like the base model. Before reaching for more data, check that the learning rate and number of epochs are reasonable and that your loss is computed only on the parts of the examples you want the model to learn.

← All Terms
ESC