pass@k
Why it matters
pass@k is behind most of the coding numbers you see in model announcements, and the idea has spread to math and reasoning benchmarks. Knowing how to read it prevents a common mistake: comparing one model's pass@1 with another's pass@10. The second number will always look better, and it means something different.
The gap between pass@1 and pass@k is informative too. A model with pass@1 of 40% and pass@100 of 90% often "knows" how to solve the problem but doesn't do so reliably. That gap is exactly what techniques such as sampling many candidates, self-checking and agent loops try to close.
Deep Dive
You can't grade generated code by comparing it to a reference answer word for word, because there are countless correct ways to write the same function. What matters is whether the code works. So code benchmarks run the model's output against unit tests, and report pass@k: let the model generate k different solutions to a problem, and count the problem as solved if any one of them passes every test.
pass@1 is the strict version: one attempt, right or wrong. pass@10 and pass@100 show what the model can do when given many tries. The metric was popularized by OpenAI's 2021 Codex paper, which introduced the HumanEval benchmark of 164 Python problems.
How It's Computed
The naive way is to draw exactly k samples and check them, but the result is noisy. The standard method draws a larger number n of samples per problem (say 200), counts how many are correct (c), and computes the probability that a random set of k from those n contains at least one correct one:
pass@k = 1 − C(n − c, k) / C(n, k)
averaged over all problems. Sampling settings matter. For pass@1, a low temperature is best, because you want the model's single most likely answer. For large k, a higher temperature helps, because you want diverse attempts.
The Catch: Someone Has to Pick
pass@k assumes an oracle that knows which of the k attempts is correct. In a benchmark, the hidden tests play that role. In real life, you get k candidate solutions and no answer key, unless you have tests of your own. This is why pass@1 is the honest number for "will this work when I ask once", and why tests are so valuable in practice: an agent that can run tests and retry effectively converts a pass@k ability into a pass@1 result.
Related Metrics
- Resolved rate: benchmarks such as SWE-bench report the share of tasks solved in one agent run, which is essentially pass@1 for a whole agent.
- pass^k ("pass hat k"): the chance that all k attempts succeed. It measures consistency, and matters for agents that must do the same job correctly every time, such as customer service bots. A model can have high pass@1 and poor pass^8.
- Majority voting / self-consistency: for tasks with a single answer, take the most common answer across k samples. No oracle needed.
Caveats
- Weak tests inflate scores. When researchers added many more tests to HumanEval, scores dropped noticeably, because some "passing" solutions were wrong.
- Old benchmarks are saturated and contaminated. HumanEval and MBPP problems are short, public and solved by modern models at well over 90%.
- Small tasks only. Passing a self-contained function test says little about working in a large codebase.