Skip to main content
Zubnet AI › Learn › Wiki › MMLU
Training

MMLU

Also known as: Massive Multitask Language Understanding, MMLU-Pro, GPQA
A multiple-choice exam of about 16,000 questions across 57 subjects that was the standard measure of an LLM's general knowledge until models maxed it out.

Why it matters

For roughly four years, MMLU was the first number in almost every model announcement and model card. It offered one easy-to-read figure for "how much does this model know?" across a broad range of fields.

It's also a textbook case of a benchmark's life cycle. GPT-4 reached about 86% in 2023, and frontier models have clustered around 90% since. At that point, differences between models are smaller than the benchmark's own noise, and the field moved on to harder tests. You'll still see MMLU quoted, especially for small and open models, where it continues to separate the weak from the strong.

Deep Dive

MMLU was published in 2020 by Dan Hendrycks and colleagues. It gathers questions from real exams and study materials in 57 subjects, from elementary math and US history to college physics, professional law and medicine. Every question has four answer choices, so random guessing scores 25%. The score is simply the percentage answered correctly.

When it came out, the best model, GPT-3, scored about 44%, while expert humans were estimated at around 90%. That wide gap made it a useful yardstick for years.

How It's Run

Models are usually given a few example questions with answers (5-shot) and then the test question, and must output a letter, A to D. Results depend on details such as the prompt format and how the answer is extracted, so scores reported by different sources aren't always comparable.

Known Problems

  • Saturation: top models are at or near the ceiling.
  • Errors in the data: a 2024 review called MMLU-Redux re-checked a sample and estimated that over 6% of questions have mistakes, such as wrong answer keys or unclear wording. In some subjects, such as virology, more than half of the checked questions were flawed. A perfect score isn't even possible.
  • Contamination: the questions have been public for years and are certainly in training data.
  • Format: multiple choice rewards recognition over generation, and a model can sometimes eliminate options without knowing the answer.

Successors

  • MMLU-Pro (2024): about 12,000 questions with ten answer choices instead of four and more emphasis on reasoning, which pulled top scores back down.
  • GPQA (2023): 448 graduate-level questions in biology, physics and chemistry, written by domain experts to be "Google-proof". Skilled non-experts with web access and half an hour per question scored only 34%, while experts scored 65%. Its hardest subset, GPQA Diamond, became a standard reasoning-model benchmark, and by 2025 leading models scored above the expert baseline.
  • Humanity's Last Exam (2025): expert-written questions designed so that the frontier models of the time failed them.

How to Read an MMLU Score

Treat it as a rough filter. A model at 60% and a model at 85% are clearly different. Two models at 88% and 90% aren't, at least not on this evidence. For choosing a model for your product, a small test set built from your own tasks tells you more than any general knowledge exam.

← All Terms
ESC