Skip to main content
Zubnet AI › Learn › Wiki › Needle in a Haystack
Training

Needle in a Haystack

Also known as: NIAH, Needle Test
A long-context test that hides one small fact inside a very long document and checks whether the model can find it.

Why it matters

The test gave everyone an intuitive picture of long-context reliability, and its heat maps spread widely. The first results showed that models did worse as documents got longer and often missed needles placed in the middle, echoing research on the "lost in the middle" effect. Model makers responded quickly, and a perfect needle chart became a standard slide in launch presentations.

Today the basic test is mostly solved: leading models retrieve a single needle almost perfectly across hundreds of thousands of tokens. That makes it important to know what the test does and doesn't prove.

Deep Dive

When models with 100,000-token context windows arrived in 2023, an obvious question followed: can they actually use all that text? The needle-in-a-haystack test, popularized by developer Greg Kamradt that November, offered a simple way to check. Take a long stretch of unrelated text (the haystack; he used essays by Paul Graham). Insert one out-of-place sentence (the needle), such as a statement about the best thing to do in San Francisco. Then ask the model a question only the needle answers.

Repeat for different document lengths and different needle positions, from the very start to the very end, and plot the results as a heat map. Green means found, red means missed.

Why Passing It Proves Little

The classic needle is easy in a specific way: it looks nothing like its surroundings, and the question shares words with it. Finding it is closer to a keyword search than to understanding. Real long-context work is harder: the relevant passage resembles everything around it, the question is phrased differently from the text, several facts must be combined, or the answer depends on the document as a whole.

Harder Versions

  • Multi-needle: several facts are hidden, and the model must retrieve all of them or reason across them.
  • No lexical overlap: benchmarks such as NoLiMa write the question so it shares no keywords with the needle, forcing the model to make an inference to connect them. Scores of many models that ace the classic test fall sharply, even at a few tens of thousands of tokens.
  • Distractors: near-miss sentences that resemble the needle but give wrong answers.
  • Reasoning over long inputs: suites such as RULER and BABILong add tracing variables, counting and aggregating across the whole context.

These versions reveal the gradual decline in quality with input length that practitioners call context rot.

Lessons From the Original Runs

Results were sensitive to prompting. After one model scored poorly, its maker showed that adding a single sentence to steer it toward the most relevant passage raised accuracy from about 27% to 98% on the original evaluation. The takeaway: long-context performance depends on how you ask, not only on the model.

Running Your Own

The idea is easy to adapt, and that's its lasting value. Build your haystack from your own documents, and make the needle realistic: a clause in a contract, a line in a log, a function in a codebase. Ask the kinds of questions your users ask. Test at the context lengths you actually plan to use. That tells you more about your application than any published chart, and it helps you decide when to pass whole documents and when to use retrieval.

← All Terms
ESC