ARC-AGI
Why it matters
ARC-AGI is built around a specific claim: intelligence is not how much you know, but how efficiently you pick up new skills. Benchmarks full of knowledge questions can be passed by memorization at scale. ARC tries to make that impossible by using novel puzzles, kept partly in private test sets.
That's why it has been central to AGI debates. Large language models that aced knowledge exams scored near zero on ARC for years. Then in December 2024, OpenAI's o3 reasoning model reached 75.7% on the semi-private set within the compute limits, and 87.5% using far more compute. That was the clearest sign yet that test-time compute and reasoning models changed what AI could do, though at a cost of thousands of dollars per task at the high end.
Deep Dive
ARC was created by François Chollet, the creator of the Keras deep learning library, in 2019 alongside his paper "On the Measure of Intelligence". Each task shows a few example pairs of small colored grids: an input grid and the output grid it should become. The solver must figure out the hidden rule (fill enclosed shapes, mirror the pattern, count objects and draw a bar) and apply it to a new input. There's no text, no trivia and no specialized knowledge, just abstraction.
Most people solve these puzzles easily. For years, AI systems did not.
The Versions
- ARC-AGI-1 (2019): the original set of 800 public tasks plus hidden evaluation sets.
- ARC-AGI-2 (2025): harder tasks designed to resist brute-force search and to need more compositional reasoning. At launch, leading AI systems scored in the low single digits while human testers solved the tasks, and scores have been climbing since.
- ARC-AGI-3 (2026): an interactive benchmark, launched on 25 March 2026, where agents must explore small game-like environments and figure out the goals and rules themselves. At launch, humans scored 100% and frontier AI systems under 1%.
The ARC Prize
Since 2024, the ARC Prize Foundation has run an annual competition with a large grand prize for reaching a high score on the private set under strict limits: solutions must run offline on modest hardware within a fixed time. That keeps the focus on efficient methods rather than throwing unlimited compute at the problem. Winning approaches have included test-time training, where a model fine-tunes itself briefly on each puzzle's examples.
Efficiency Is Part of the Score
The leaderboard reports cost per task alongside accuracy. Reaching a score with $10,000 of compute per puzzle is treated as a very different achievement from reaching it for a few cents. This reflects the benchmark's core idea that intelligence includes efficiency.
Reading ARC Results
A high ARC score is meaningful evidence of flexible reasoning, but it isn't proof of general intelligence. The puzzles are one narrow format, and they don't test language, real-world knowledge or long-horizon planning. Chollet himself has said that "passing ARC-AGI does not equate to achieving AGI", and ARC Prize president Greg Kamradt has called beating ARC necessary but not sufficient for AGI. Treat it as one of the hardest and most honest reasoning tests available, not as a finish line.