Skip to main content
Zubnet AI › Learn › Wiki › SWE-bench
Training

SWE-bench

Also known as: SWE-bench Verified
A benchmark that tests whether AI systems can resolve real GitHub issues by editing real codebases until the project's tests pass.

Why it matters

SWE-bench became the headline benchmark for coding agents, and it was quoted in almost every major model launch in 2024–25. It's popular because it's close to real work: understanding an unfamiliar codebase, finding the right files, making a change that fits the project, and not breaking anything else.

Its trajectory also tracks the rise of agents. When it launched, the best systems solved under 2% of tasks. By 2025, top models with good agent harnesses solved well over 70% of SWE-bench Verified. That jump came from both better models and better scaffolding: tools for searching, reading files, running tests and iterating.

Deep Dive

SWE-bench was released in 2023 by researchers at Princeton. Instead of asking a model to write a small function from scratch, it hands the model a real bug report or feature request from a popular open-source Python project, along with the full repository at the time the issue was filed. The model must produce a code change. The change is then checked by running the tests that the project's real developers wrote when they fixed that issue.

The original set has 2,294 tasks from 12 repositories, including Django, scikit-learn, sympy and matplotlib. Its best-known version, SWE-bench Verified, is a 500-task subset that OpenAI and the SWE-bench authors released in 2024 after human engineers screened out tasks that were ambiguous or had unfair tests.

How a Run Works

A typical agent run: the agent reads the issue, explores the repository with search and file tools, reproduces the bug, edits code, runs tests and iterates. When it's done, its final patch is applied to a clean copy of the repository, and two kinds of tests run: the "fail-to-pass" tests that should now pass, and "pass-to-pass" tests that must not break. Both must succeed for the task to count.

Variants

  • SWE-bench Lite: 300 tasks, chosen to be cheaper and somewhat simpler.
  • SWE-bench Verified: the 500 human-validated tasks, which became the standard comparison. In February 2026 OpenAI stopped reporting it, citing contamination and flawed tests, and now points to SWE-bench Pro.
  • SWE-bench Multimodal: issues that include screenshots or visual bugs, mostly from JavaScript projects.
  • Multilingual and "Pro" variants: newer efforts to cover more languages, longer tasks and repositories less likely to be in training data.

Caveats Worth Knowing

  • Scaffolding matters: the same model can score very differently depending on its agent harness, tools and number of attempts, so compare numbers only when setups match.
  • Contamination: the repositories and their fixes are public, so models may have seen them during training.
  • Narrow slice: it's mostly Python, mostly bug fixes, mostly one well-known set of projects. A great score doesn't guarantee great results on your private codebase.
  • Saturation: as top scores climb, differences at the top shrink and become noisy, which is why harder successors keep appearing.

How to Use It

Treat SWE-bench as a strong signal of agentic coding ability, not a final answer. The best test is still a handful of real tasks from your own repositories, run with the tools you'll actually use.

← All Terms
ESC