A 99% needle-in-a-haystack number is bought by literal word overlap between the question and the planted fact. Strip the overlap, or make the haystack realistic, and the same model collapses. NIAH does not predict long-context ability.
A model ships with a marketing graph almost too clean to read. It is the needle-in-a-haystack chart: a context window stretched to 128K tokens, a planted sentence the model is asked to retrieve, and a wall of green squares saying it found that sentence every time, at every depth and every length. The number underneath is 99-point-something, and the message is unmissable — this model handles long context. A team picks the model partly on the strength of that chart, points it at a 100K-token workload, and waits for the green wall to show up in production.
It does not show up. The model that scored 99% misreads documents two-thirds of the way down a long context, follows the wrong thread when several are present, and loses the plot once the context is a real pile of retrieved passages rather than a clean essay with one sentence hidden in it. Nothing was faked — the 99% was a real measurement. The team simply read it as a statement about long-context ability, and it was not one.
A vanilla needle-in-a-haystack score measures one thing: can the model find a sentence that shares words with the question, somewhere in a block of unrelated filler. That is a lexical-overlap retrieval task, near-trivial once the overlap is there. Long-context ability is a different construct — reasoning across a full window, tracking multiple threads, surviving distractors, using a fact that does not announce itself with a matching keyword. A stack of recent benchmarks exists precisely to measure the distance between the two, and they find it large.
This post is about why a passing NIAH score is not evidence of long-context ability: why the headline number is bought by literal lexical overlap, why it collapses the moment you remove that overlap or make the haystack realistic, and why it does not predict how the model behaves on the long-context work you actually have.
The construct error sits so close to “long context” in everyone’s mental model that it feels like a direct measurement of it. Vanilla NIAH plants a sentence — the needle — inside a long stretch of unrelated text and asks a question whose answer is in the needle. The catch is in how the two relate. In the standard benchmarks they share words, heavily, and the model can find the needle by doing the easiest thing a language model does: spotting the span whose tokens match the query’s tokens. NoLiMa: Long-Context Evaluation Beyond Literal Matching (arXiv 2502.05167) states the loophole flatly — in these benchmarks, models can exploit existing literal matches between the needle and the haystack to simplify the task. The retrieval is real, but it is surface retrieval: it does not require understanding the long context, only pattern-matching a string.
NoLiMa takes that crutch away. It builds a needle set where the question and the needle have minimal lexical overlap, so the model must infer a latent association instead of matching words. The size of the gap it removes is the tell. By surface overlap between question and needle, vanilla NIAH scores a ROUGE R-1 of 0.905 — almost the same words. NoLiMa scores 0.069 — nearly disjoint. That one number is the difference between “find the sentence that repeats your question” and “find the sentence that answers it without repeating it,” and the paper notes that direct examples with a high degree of literal overlap are straightforward for the model to answer even in long contexts. Strip the overlap, and you find out what the model was actually capable of.
If the 99% were measuring long-context ability, removing the word overlap would barely move it. It does not work out that way. NoLiMa evaluated 13 popular LLMs that all claim to support contexts of at least 128K tokens. Under 1K tokens they perform well, but as the context grows the floor falls out. At 32K, 11 of the 13 drop below 50% of their own strong short-length baseline: not below 50% absolute, but below half of where they started, which controls for how good each model is to begin with. The single number that should retire the marketing chart is GPT-4o’s. One of the top performers, it still falls from an almost-perfect 99.3% baseline to 69.7% once the overlap is gone and the context is long. That 99.3% is the green wall the chart would print; the 69.7% is what remains when the task stops being a string match — thirty points that were the overlap, not the model. The rest of the picture at 32K is worse: Gemini 1.5 Pro at 48.2%, Llama 3.3 70B at 42.7%, Claude 3.5 Sonnet at 29.8% — the models the NIAH charts celebrated, measured on a task that asks for understanding instead of matching.
The obvious objection is that reasoning should rescue this; the data says no. On the harder NoLiMa subset, Llama 3.3 70B with CoT prompting scores 31.8% at 16K and 10.1% at 32K, and even a dedicated reasoning model, GPT-o1, drops to 31.1% at 32K. Reasoning helps at the margins and then falls off the same cliff, because the bottleneck is the model’s grip on the long context itself, not the depth of its reasoning over a fact it has already found.
Changing the haystack breaks the score for a different reason — two independent failure modes are harder to dismiss as one benchmark’s quirk. Vanilla NIAH uses a homogeneous haystack: one planted needle inside filler that has nothing to do with it. Real long contexts do not look like that. They are assembled by retrievers and agents, full of passages topically near the needle — near-misses a dense retriever surfaced, or distractors a multi-step agent generated itself. Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluation (arXiv 2510.07414) notes that synthetic NIAH tests overlook exactly this — how noisy contexts arise from biased retrieval and agentic workflows — and builds HaystackCraft on the full English Wikipedia hyperlink network with multi-hop questions to put it back.
The numbers echo the NoLiMa collapse from a new direction. With a clean, distractor-free haystack the models do fine; once a real dense retriever fills it with topically-similar distractors, the same models fall apart at 128K. Llama-3.1-70B-Instruct drops from 67.7 with no distractors to 25.85, Llama-3.1-8B-Instruct from 59.8 to 31.51, Qwen2.5-7B from 55.56 to 36.14 — all suffer significant performance degradation as the context grows to 128K, regardless of the retrieval strategy. The needle was always present and findable; what changed was the company it kept, and a NIAH chart never shows this because a NIAH haystack has no company in it.
The agentic setting is worst. When the model operates in rounds — generating, retrieving, reasoning over its own outputs, the way an agent actually runs — the best frontier models cascade-fail on distractors they generated themselves. Even Gemini 2.5 Pro and GPT-5 are vulnerable to cascading errors, and none of the models reliably improve upon their single-round performance. A model can post a perfect single-shot NIAH score and still degrade every round it runs as an agent. The static chart measures one shot in a clean room; the agent runs many shots in a room it is filling with its own noise.
So far NIAH breaks under two specific stresses. The stronger claim — that the score does not predict long-context performance at all — is made by the benchmarks built to test that prediction.
HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly (arXiv 2410.02694) replaces noisy synthetic signals with seven application-centric categories and evaluates 59 long-context LMs at lengths up to 128K. Its headline finding is this post’s thesis: synthetic tasks like NIAH do not reliably predict downstream performance. The mechanism is a saturation gap — most frontier models achieve effectively perfect NIAH scores, yet their behavior on real tasks diverges sharply because the diverse categories exhibit distinct trends and low correlations with each other. A metric pinned at its ceiling while the capability spreads underneath has stopped carrying information, and since the categories rank models differently, no one number can stand in for long-context ability. The difficulty NIAH never touches is where models separate: where tasks require full-context reasoning or complex instructions, the open-versus-closed gap widens with length.
RULER: What’s the Real Context Size of Your Long-Context Language Models? (arXiv 2404.06654) makes the same point and adds the context-length claim it underwrites, calling the vanilla NIAH test indicative of only a superficial form of long-context understanding. Evaluating 17 models that all claim 32K-or-greater context, it finds that despite achieving nearly perfect accuracy in the vanilla NIAH test, almost all exhibit large performance drops as the context grows — of those 17, only half can maintain satisfactory performance at 32K. The claimed context length a NIAH chart is used to justify is roughly double the effective one for half the field: a statement about where the model can still match a string, not where it can still think.
The independent confirmations stack up. 100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability? argues NIAH-like benchmarks rely on purely synthetic contents that do not reflect how LLMs are used in real-life scenarios, and — the deeper problem — that standard benchmarks evaluate long-context capabilities solely based on task scores without isolating the influence of a model’s baseline abilities. A high score may mean the model is good at long context or merely good in general, and the metric cannot tell you which. Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks? (arXiv 2411.05000) adds the same verdict across 17 leading LLMs: for many the effective context limit is significantly shorter than the supported context length, with accuracy decreasing as the context window grows. Even on tasks close to NIAH’s own genre, the advertised length overstates the usable one.
The version of this argument that overstates loses the reader who has run these benchmarks, so two qualifications.
First, NIAH is not measuring nothing. NoLiMa’s own models perform well under 1K tokens even with the overlap removed; short-range retrieval is genuinely solid. A NIAH pass is a true statement that the model can do surface retrieval at short range — the error is reading that narrow truth as the broad one. The score is not fraudulent; it just answers a smaller question than the chart implies, one modern models ace.
Second, the collapse is not uniform, and the headline numbers are not the absolute floor. GPT-4o keeps 69.7% at 32K on NoLiMa — degraded from 99.3%, but above most of the field — and some HaystackCraft models hold more of their clean score than others. The claim is not that long context is impossible; it is that the NIAH number, saturated for every model, does not tell you where on that spread a given one sits. HELMET’s low cross-category correlations cut both ways: a strong NIAH score does not promise strong downstream behavior, and a model weak on one realistic category may be fine on another. The score is real but narrow, the degradation real but uneven, and the only way through both is to stop reading one saturated number as a summary of a capability it does not summarize.
You cannot make a NIAH score predict long-context performance by wanting it to. You can stop letting it stand in for the capability, and measure the capability directly.
Treat a NIAH pass as a precondition, never a conclusion. A perfect needle score earns the right to be evaluated further; it does not earn a “handles long context” claim. The green wall is the floor of the evaluation, not the ceiling.
Remove the lexical crutch. If you must run a needle test, run one where the question and the needle do not share words — the NoLiMa shape — so the model has to infer the association rather than match the string. The gap between a model’s high-overlap and low-overlap scores is the portion of its headline number that was free.
Test on a realistic, heterogeneous haystack at the length you serve. Fill the window with topically-similar near-misses — the HaystackCraft shape — and measure at the length the workload runs. A needle pass at 4K does not certify behavior at 100K, and 100K is where you turned long context on.
If the model runs as an agent, measure it round by round. Single-shot retrieval and multi-round agentic operation are different tasks, and the frontier models that ace the first cascade-fail on the second over their own distractors. The robustness story is the same as why an agent’s benchmark score is not its reliability: the curated number is a good-day measurement, and the realistic distribution is harder.
Score multiple downstream categories, and separate them from baseline ability. HELMET’s categories correlate weakly, so a model good at one may be weak at another; evaluate the long-context tasks the product depends on — full-context reasoning, multi-hop synthesis, instruction-following across the window — as their own lines, never averaged into a single number. A high score can also just be a model that is good in general, so where you can, use a length-controllable evaluation that isolates the long-context delta from the short-context baseline.
Before you let a needle-in-a-haystack score back a “handles long context” claim:
The green wall is a true measurement of a small thing: the model can find a sentence that repeats your question, in a clean block of filler, at the tested length. It was never a measurement of the large thing the chart is used to sell — reasoning across a full, messy, realistic context. The two come apart where it matters: remove the word overlap and a 99.3% becomes 69.7%, fill the haystack with real distractors and the scores halve again, and the benchmarks built to predict downstream use say plainly the needle number does not predict it. Treat a NIAH pass as the entry fee, not the verdict, and measure the long context you actually have. It is the same lesson as retrieval recall not being answer quality: the clean component number and the system you ship run on separate clocks.
Teams optimize the retriever — chase recall@k, retrieve more documents — and expect answer quality to follow. It does not. Recall and answer accuracy are decoupled, and past a small number of documents, more retrieved context routinely makes the answer worse.
16 min →
Heuristic privacy defenses report safety by beating weak attacks on average-case samples. Audit them honestly and they leak about 10x more — and lose to the provable DP-SGD baseline they never ran, even one whose formal epsilon is vacuous.
13 min →
RAG corpora ingest documents from sources no one fully trusts, on the assumption that a wrong document is just noise the retriever averages out. It is not noise. A few crafted passages — sometimes one — reliably flip the answer to a targeted question.
14 min →