Skip to main content
Zubnet AI › Learn › Wiki › Cross-Entropy
Fundamentals

Cross-Entropy

Also known as: Cross-Entropy Loss, Log Loss, Negative Log-Likelihood
The loss function nearly every language model is trained on, which penalizes the model according to how little probability it gave to the correct answer.

Why it matters

When you watch a training run and see "loss: 2.31", that number is almost always cross-entropy. It's the quantity pre-training and supervised fine-tuning push down, and the main signal for diagnosing overfitting and underfitting.

It also explains model behavior. Because the log punishes confident mistakes severely, models learn to spread probability across plausible options rather than bet everything on one. That calibrated uncertainty is what makes sampling, temperature and top-p work.

Deep Dive

A language model doesn't output one word. It outputs a probability for every token in its vocabulary. Cross-entropy turns that into a single score by asking one question: how much probability did you put on the token that actually came next? The loss is the negative logarithm of that probability. Give the right token a probability of 1 and the loss is 0. Give it 0.5 and the loss is about 0.69. Give it 0.01 and the loss is 4.6.

Average that over billions of tokens and you have the training objective of GPT, Claude, Llama and essentially every other LLM.

Reading the Number

Cross-entropy is measured in nats (natural log) or bits (log base 2). A few anchors help:

  • A model guessing uniformly over a 50,000-token vocabulary has a loss of ln(50,000) ≈ 10.8.
  • A loss of 2.0 means the model gives the correct token about 13.5% probability on average (e⁻²).
  • Perplexity is just e raised to the loss. A loss of 2.0 is a perplexity of about 7.4: the model is as uncertain as if it were choosing among roughly 7 equally likely tokens.

The loss can never reach zero on real text, because language is unpredictable. Nobody can know for certain which word comes next.

Why This Loss

Minimizing cross-entropy is the same as maximizing the likelihood of the training data, a principle statisticians have used for a century. It also pairs neatly with softmax: the gradient with respect to the logits is simply the predicted probabilities minus the true answer, which is cheap and numerically stable. That's why frameworks combine the two into a single function that takes raw logits.

Variations

  • Binary cross-entropy: the two-class version, used for yes/no classifiers and multi-label problems.
  • Label smoothing: replaces the hard target with a slightly softened one (say, 90% on the right answer) to reduce overconfidence.
  • KL divergence: a close cousin measuring the gap between two distributions. Distillation trains a student to match a teacher's full output distribution this way.
  • Masking: in fine-tuning, the loss is usually computed only on the response tokens, not the prompt.

What It Doesn't Measure

A lower cross-entropy means better next-token prediction, not necessarily a more helpful or truthful model. Two models with similar loss can behave very differently in conversation. That's why loss curves are paired with task evaluations and human review, and why later training stages use preference or reward signals instead.

← All Terms
ESC