Skip to main content
Zubnet AI › Learn › Wiki › Humanity's Last Exam
Training

Humanity's Last Exam

Also known as: HLE
A benchmark of 2,500 expert-written questions at the edge of human knowledge, built in 2025 specifically so that the best AI models of the day would fail it.

Why it matters

HLE reset the scoreboard. At launch, the top models scored in the single digits: GPT-4o around 3%, and the best reasoning models of early 2025 under 15%. That gave the field room to measure progress again.

Progress came fast. Within months, reasoning models with search tools and deep research agents passed 20%, and by late 2025 the best systems passed 40%: Gemini 3 Deep Think scored 41% without tools, and systems with tools scored higher on the text-only subset. The benchmark became a regular line in frontier model announcements, and it now has successors, HLE-Rolling and HLE-Diamond. Its trajectory is one of the clearer signals of how quickly expert-level question answering is improving.

Deep Dive

By 2024, AI models were scoring around 90% on the standard knowledge benchmarks, which made those tests useless for telling the best models apart. Humanity's Last Exam was the response. Organized by the Center for AI Safety and Scale AI and released in early 2025, it collected questions from nearly a thousand subject experts, mostly professors and researchers, at hundreds of institutions worldwide.

The key rule: a question was accepted only if the leading AI models got it wrong. Contributors competed for a $500,000 prize pool for the best submissions.

What the Questions Look Like

The 2,500 questions span more than a hundred subjects, with math the largest share, followed by biology and medicine, computer science, physics, the humanities, chemistry and engineering. Examples include translating an ancient inscription, counting specific anatomical features of a particular bird family, or solving a research-level math problem. Most require a short exact answer, about a quarter are multiple choice, and a minority include an image. Each has one verifiable answer that can't be found with a quick web search.

Calibration

HLE also asks models to state their confidence. Early results showed severe overconfidence: models gave wrong answers while claiming high certainty, with calibration errors above 70% or 80%. This makes HLE a hallucination test as well as a knowledge test. A model that knows what it doesn't know is more useful than one that guesses confidently.

Criticisms

  • Trivia or intelligence? Many questions test extremely obscure knowledge. Scoring well shows breadth and careful reasoning, but it isn't the same as doing a researcher's job.
  • Answer quality: questions at the frontier are hard to verify. An independent 2025 review argued that a sizable share of the chemistry and biology answers conflicted with published literature. The maintainers have corrected and replaced questions over time.
  • Adversarial selection: questions were filtered against specific 2024 models, so early scores partly reflected that filter, not just difficulty.
  • Tools change the test: a model with web search is doing a different task than one without, so results should say which was used.
  • The name: it won't be the last exam. Harder benchmarks, and ones based on long, open-ended tasks, are already following.

Reading the Results

Compare like with like: same tool access, same reasoning effort. Look at calibration alongside accuracy. And remember that the public questions will leak into training data over time; the organizers keep a private set to detect models that have overfit to the public one.

← All Terms
ESC