A leaderboard row reads as a property of the model, but it is jointly produced by the model and the undisclosed execution harness. Swapping the harness on a fixed model moves the score far more than the model does — often enough to reverse the ranking.
A team picks a model for an agent the way everyone does: it reads the leaderboard. Two frontier models, a coding-agent benchmark, a gap of a few points between them. The team takes the model on top, wires it into their own harness — their context construction, their tool definitions, their orchestration loop, their verifier — and ships. The agent underperforms the number that sold it. Not catastrophically; it just does not reproduce the leaderboard. The team assumes they wired something wrong, tunes the prompt, swaps the model order, and the relative standing of the two models flips against the published table. The model that was supposed to be second is now first.
Nothing is broken. The team has run head-first into a measurement error built into how agent benchmarks are reported. The leaderboard number was never a property of the model alone — it was produced by the model and the harness around it, and the harness the benchmark used is not the one the team built, was almost certainly not disclosed, and is not even the one used for the row directly above or below on the same table. The mistake is reading a model-harness result as a model result. A long-horizon agent’s score is a joint measurement of two things, and only one is the thing the leaderboard claims to rank. The other — the scaffold — is usually larger, almost always hidden, and rarely held constant across the rows you compare.
This post is about why an agent benchmark number is mostly the harness, why swapping the harness on a fixed model can move the score by many points and reverse rankings, why those swings dwarf the deltas papers report as model progress, and what it would take to make the comparison mean something.
Start with the operation a benchmark actually performs, because the standard mental model quietly drops a variable.
The intuitive picture is that a benchmark measures a model: point the suite at Model A, get A’s score; point it at Model B, get B’s; compare. That is right for a static benchmark — a question set scored against a key, where the only moving part is the model’s answer. It is wrong for an agent benchmark, because between the model and the task sits a layer that does an enormous amount of the work: the harness. The harness decides what context the model sees, which tools it can call and how they are described, how the loop is orchestrated and retried, and how a result is verified. The model is one component inside a closed loop the harness controls.
So the leaderboard number is f(model, harness), not f(model) — two free variables, one reported. And the unreported one is not a correction term. From Model Scaling to System Scaling: Scaling the Harness in Agentic AI (arXiv 2605.26112) describes the regime: agent performance “emerges from the interaction among the foundation model, memory substrate, context constructor, skill-routing layer, orchestration loop, and verification-and-governance layer.” Yet evaluation, it argues, “remains largely model-centric, often reducing agents to final-task success while treating memory, retrieval, tool use, orchestration, verification, and governance as secondary implementation details.” The harness is a free variable held in the experimenter’s hand and not written down.
Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows (arXiv 2605.27922) draws the conclusion that follows. Across 5,194 execution trajectories spanning representative harness configurations and multiple model backends under shared tasks, budgets, and protocols, it finds “substantial variation in completion, process quality, efficiency, and failure behavior across model-harness pairings,” and states the corrective directly: agent capability “should be reported at the model-harness configuration level rather than attributed to the base model alone.” Capability is a property of the pair, so a number credited to the model alone is mis-attributed by construction.
If the harness is a free variable, the test is direct: hold the model fixed, change only the harness, and see how far the score moves. If it barely moves, the harness is a detail and the leaderboard is fine. It does not barely move.
Stop Comparing LLM Agents Without Disclosing the Harness (arXiv 2605.23950) runs the controlled factorial — same models, same tasks, harness varied as a deliberate factor — and reports the variances side by side. Average harness-induced variance is 18.48 pp² versus average model-induced variance of 2.37 pp², a ratio of 7.80×. The scaffold moves the score roughly eight times as much as the model does: change which model you run and the score wobbles; change the harness around a fixed model and it lurches.
The single-instance version is more vivid. On SWE-bench Pro, holding the model fixed and changing only the harness, the same model — Claude Opus 4.5 — reaches 45.9% under the standardized SEAL scaffold and 55.4% under Claude Code. One model, one benchmark, two harnesses, about 9.5 points of spread. Only the layer the leaderboard does not print changed.
Hold that 9.5 against what agent papers report as a generation of model progress. On Randomness in Agentic Evals (arXiv 2602.07150) — 60,000 agentic trajectories on SWE-Bench-Verified, three models, two scaffolds — finds even the run-to-run noise floor is large: single-run pass@1 estimates “vary by 2.2 to 6.0 percentage points depending on which run is selected, with standard deviations exceeding 1.5 percentage points even at temperature 0,” and concludes that “reported improvements of 2–3 percentage points may reflect evaluation noise rather than genuine algorithmic progress.” Stack the three up: model progress is reported in 2–3 point deltas, run-to-run noise alone covers 2.2–6.0 points, and a harness swap on a fixed model is worth about 9.5. The delta the leaderboard wants you to read is the smallest of the three, reported against two larger sources of variation — one usually a single run, one undisclosed.
A skeptic can grant all of that and still hold the line that matters: fine, the harness shifts the absolute numbers, but if it shifts everyone similarly the ranking survives, and ranking is what a leaderboard is for. That defense is the one the evidence specifically kills.
The same factorial study reports ranking reversals: 6 out of 9 model-pair/harness-pair comparisons reverse. In a majority of head-to-head cases, which model ranks higher depends on which harness ran them. The ordering is not a stable fact the harness merely scales — it is a joint outcome the harness can flip. “Model A beats Model B” is not a complete sentence for a long-horizon agent; “Model A beats Model B under this harness” is, and the qualifier is the part nobody prints.
The mechanism is not exotic, and the paper pins it to a component you would never think to disclose: adding a single search subagent — WarpGrep — to otherwise identical infrastructure flips the SWE-bench Pro ordering between MiniMax 2.5 and Claude Opus 4.6. One subagent, and the model that was behind is now ahead — not different weights, not more training, one piece of scaffolding. If a published ordering can be inverted by a harness component the leaderboard does not name, the ordering is a claim about the harness wearing the model’s clothes.
This is the seam that separates this failure from its neighbors. It is not that the score fails to survive distribution shift — the gap between a benchmark score and production reliability is a real and separate problem, but it is about the number degrading out of distribution. This is worse and stranger: even on the fixed benchmark, in-distribution, with the data held constant, the reported number is mis-credited. The model is not the thing that produced the ranking. The undisclosed scaffold is.
There is a tempting way to minimize all this: treat the harness as a thin, near-standard wrapper any two setups share to within a rounding error. If that were true, “model-harness pair” would collapse back toward “model,” because the harness term would be roughly constant. The size of the design space is why it is not.
Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures (arXiv 2604.03515) reads the source of 13 open-source coding-agent scaffolds at pinned commit hashes and finds “scaffold architectures resist discrete classification: control strategies range from fixed pipelines to Monte Carlo Tree Search, tool counts range from 0 to 37, and context compaction spans seven distinct strategies,” with 11 of 13 composing multiple control primitives. A harness whose tool interface alone ranges from zero to thirty-seven tools is not a shared constant; two leaderboard rows almost never run the same one, so “same model, different harness” is a comparison across genuinely different systems. And this is the part nobody studies — the scaffolding code “remains poorly understood” even though it determines behavior, so the variable with the largest effect on the score is the one the field has characterized least. That the tool interface lives here is not incidental: the tool descriptions are themselves a ceiling on tool-calling reliability, and they are a harness property, not a model one — two of the largest known levers on agent performance both sit in the layer the leaderboard omits.
It compounds because the harness does not just wrap the task — it can change it. OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Coding (arXiv 2601.10343) instantiates 217 tasks across three scaffold types — 34 environments, 7,098 objective checklist items — and finds, across eight models, “a systematic gap between task-solving and scaffold-aware compliance.” The scaffold specifies instructions the model must follow, so the same task under a different scaffold is, in part, a different problem, and a single success number conflates solving the task with obeying the scaffold’s rules. The harness is not a passive pipe capability flows through unchanged; it is part of the question.
If the harness is this load-bearing, the obvious response is to demand it — read each row’s harness and re-attribute. You usually cannot, and the reason is structural.
What Twelve LLM Agent Benchmark Papers Disclose About Themselves (arXiv 2605.21404) audits 12 benchmark papers — 8 agent-focused, 4 classical static — scoring the disclosure of an agent run rather than its correctness, and finds “none fully disclose a content-addressed container image of the evaluation environment.” Zero of the audited agent benchmarks ship a reproducible image of the environment the agent ran in; the harness is not merely under-described but, at the level of reproducibility, structurally absent. The audit puts a number on it: agent benchmarks score a mean of 0.38 out of 1.0 on disclosure against 0.66 for classical static ones — the results that depend most on an undisclosed execution layer disclose least about it.
The audit is careful about its own claim: it scores disclosure, “not its correctness,” and is explicit that full disclosure would not certify a result. But the converse is the load-bearing direction — the absence of disclosure makes the number uninterpretable, because you cannot tell how much is the model and how much is the scaffold you were never shown. You are asked to read f(model, harness), credit it entirely to model, and not told what harness was.
The argument is sharp, and a sharp argument is the easiest to push past its evidence — so here is where it stops.
It does not say the model never matters. Model-induced variance was 2.37 pp² — real and nonzero; on a fixed, disclosed harness, a better model is still a better model, and the ranking under a single pinned scaffold is a legitimate measurement of that pair. The claim is about cross-model leaderboards with undisclosed, non-constant harnesses: there, the harness dominates and the model delta is mis-credited. It is not that model choice is irrelevant.
It does not say every benchmark is equally compromised. The strongest single-instance numbers — the 9.5-point spread, the reversals, the WarpGrep flip — come from coding-agent, long-horizon settings, and the Binding Constraint Thesis is stated for “long-horizon tasks evaluated across models with comparable frontier capability.” A short, single-tool task with a thin and genuinely shared harness has less room for the scaffold to dominate; the confound scales with how much work the harness does, and on a long-horizon task it does a lot.
And the magnitudes are anchored to particular setups — the 7.80× ratio, the 45.9-versus-55.4 split, the 6-of-9 reversals are measurements on specific models and benchmarks, not a constant to paste onto your own stack. The transferable claim is the shape: capability is a model-harness joint property, the harness term is large and undisclosed, and a cross-model comparison that ignores it is measuring the scaffold. The numbers are the existence proof, not the law.
You cannot make the harness disappear, and you usually cannot recover the one a leaderboard used. What you can do is stop reading a joint measurement as a model measurement, and refuse to let an undisclosed scaffold pick your model for you.
Read every agent leaderboard number as f(model, harness). Before treating a row as a fact about the model, ask which harness produced it — and if the answer is not on the page, the row is a fact about an undisclosed system, not about the model.
Refuse to compare rows whose harnesses are not pinned and identical. A delta between two models is interpretable only if the harness was held constant and disclosed for both; when it was not — the common case — the ranking can be a harness artifact, since a majority of head-to-head orderings have been measured to reverse under a harness swap.
Size the model delta against the harness swing and the noise floor, not against zero. A 2–3 point gain reported on top of a 2.2–6.0 point noise band and a ~9.5 point harness effect is not evidence of progress; it is inside error bars the leaderboard did not draw.
Benchmark on your own harness, because that is the system you ship. Capability is a property of the pair, so the only number that predicts your production behavior is your model measured inside your scaffold. Rank the candidates on that; the published row is, at best, a result for someone else’s system.
Pin and disclose your own harness as a versioned artifact. Treat context construction, tool definitions, orchestration, and verification as a first-class object you version and freeze, and hold it constant across any model comparison — a harness change is itself a change to evaluate, because swapping one subagent has flipped a ranking. This is the same discipline as treating a prompt or model swap as a deploy that needs a canary and a rollback path: the scaffold is part of what you ship, and changing it changes the score.
When you publish a number, report the configuration, ideally a reproducible image. Zero of the audited agent benchmarks shipped a content-addressed image of the evaluation environment, and that absence is what makes their numbers uninterpretable. Name the harness, version it, and ship the environment image where you can — a number without its harness is one nobody, including you in six months, can attribute.
Before you let an agent leaderboard pick your model:
The leaderboard sold a model and delivered a system. An agent benchmark number is jointly produced by the model and a harness — the context, the tools, the orchestration, the verifier — and the harness is the larger term, the one that can reverse the ranking, and the one nobody printed. The fix is not a better model off the same table; it is to stop reading a two-variable measurement as one. Pin the harness, disclose it, compare models inside the scaffold you will actually ship, and size the model delta against the swings around it. Until the harness is held constant, a cross-model agent leaderboard is ranking scaffolds, and the model on top is just the one that happened to be wearing the better one.
A membership-inference attack that separates your book from a held-out one is usually reading the calendar, not the weights. A successful attack is not a sound proof — and the null it would need to test against cannot be sampled.
17 min →
Unlearning promises a model can forget specific data without a full retrain. What it delivers is output suppression: a small benign relearning set — or simply quantizing the model — brings the 'forgotten' knowledge back.
14 min →
The EU AI Act requires AI output to be marked in a robust, reliable way. A formal impossibility theorem says strong watermarking cannot exist. Five words in the statute carry the collision.
13 min →