An LLM judge does not just have soft biases you average out — it has content-free inputs that force a 'correct' verdict on demand. A lone colon or raw token gibberish flips judges and reward models at roughly 80-100% rates: a judge used as a reward or gate is exploitable, not noisy.
A team wires an LLM judge into a reinforcement-learning loop. The setup is the responsible one the reference-based RLVR papers describe: each rollout produces an answer, the judge reads it against a known reference and returns a verdict — correct or not — that becomes the reward the policy is trained on. Months of compute will climb the gradient that judge defines. The team has done the calibration work, too — measured the judge’s position bias, randomized answer order, anchored its rubric, pinned its version. By the standards of the field, this is a judge you are allowed to trust.
Now hand it an answer that is a single colon — the character : and nothing else. A large fraction of the time, across math and reasoning benchmarks, the judge returns correct. Swap the colon for “Thought process:” and the rate climbs higher. The policy being trained does not need to learn to solve anything; it needs to learn to emit a colon, and the reward is there for the taking.
This is a different failure from the one most teams have internalized. The familiar story about an LLM judge — the subject of a prior Field Notes post — is that the judge is a model with soft, statistical biases: it favors the longer answer, the first answer, its own family’s answer, by a margin you can probe, average over, and partly calibrate away. That story is true and it is not this one. This post is about a harder class: the judge has content-free attack inputs — superficial strings with zero task content — that force a false-positive verdict not on the margin but on demand, at rates from roughly 80% to nearly 100% across frontier judges and reward models. A bias, you calibrate away. An on-demand false positive, you cannot; you retrain or guard the judge, or you stop trusting it to gate anything — and this hits a judge used as an RL reward and one used as a benchmark or deploy gate alike.
Start with the result that names the title. One Token to Fool LLM-as-a-Judge (arXiv 2507.08794) studies generative reward models in exactly the reference-based RLVR setting where a judge supplies the training signal, and finds them “systematically susceptible to reward hacking.” The mechanism is a set of inputs the authors call master keys: “superficial inputs … such as non-word symbols (e.g., ’:’ or ’.’) or generic reasoning openers (e.g., ‘Thought process:’ or ‘Let’s solve this problem step by step.’), can consistently elicit false positive rewards without any substantive reasoning.” The input carries no answer, and the judge — whose whole job is to certify an answer correct against a reference — certifies it anyway. Not occasionally: the false-positive rates are near-total on strong open judges. A single blank space reaches a measured peak of 95.1% on LLaMA3-70B-Instruct over AIME 1983-2024; “Thought process:” gives 88.7% on MATH; a bare colon : gives 90.8% on GSM8K; the word “Solution” gives 88.6% on MATH. These run across five benchmarks — Multi-subject RLVR, NaturalReasoning, GSM8K, MATH, AIME — and a range of judges, so the colon is not a quirk of one model on one dataset. Nor does the failure stop at open models, which is what matters for a team that picked a strong proprietary judge precisely because it is the trustworthy option: the paper reports “a widespread failure affecting a diverse range of models, including leading proprietary systems such as GPT-o1 and Claude-4.” A green calibration report tells you about a judge’s biases on substantive answers; it tells you nothing about what it does when handed a colon.
It would be comfortable to read the colon as a prompt-engineering accident — a phrasing that trips one judge, patchable by a better grading prompt. The follow-up work closes that exit. AdvJudge-Zero (arXiv 2512.17375) locates the failure in the judge’s structure: “their binary verdict reduces to a single linear readout F_gap on one hidden state.” A decision that collapses to a single linear readout on one activation is one a small, content-free token sequence can steer — which is why it generalizes past any hand-picked string. AdvJudge-Zero turns that into an automatic attack, sampling the tokens from the judge’s own next-token distribution — no gradients, no curated seed set, no access to anything but the judge’s outputs — and it beats the curated keys it automates, reaching “>90% ensemble false-positive rate on 22 of 24 (model, dataset) cells across six Qwen, Llama, and Gemma judges, versus 54-72% for the prior curated 10-token benchmark.” Those tokens are “patterns that a policy model could plausibly generate during post-training, and thus represent realistic reward-hacking risks rather than worst-case adversarial strings” — not a suffix an external attacker injects, but the kind of string an optimizing policy, climbing the reward the judge defines, stumbles into and exploits. The reward-hacking risk is endogenous to the training loop.
And this is not a new-models problem: well before the master-key paper, Is LLM-as-a-Judge Robust? (arXiv 2402.14016), the first study of assessment-LLM adversarial robustness, showed “short universal adversarial phrases can be concatenated to deceive judge LLMs to predict inflated scores” such that “irrespective of the assessed text, maximum scores are predicted” — and that the phrase, optimized on a surrogate, “transferred to unknown judge-LLMs,” so the attacker never needs access to the deployed judge.
The other half is the judge used as a benchmark — the automatic leaderboard a model is ranked on, the gate a release has to clear — and that side has its own content-free exploit, starker because the attack is a single fixed string. Cheating Automatic LLM Benchmarks (arXiv 2410.07137) shows that “even a ‘null model’ that always outputs a constant response (irrelevant to input instructions) can cheat automatic benchmarks and achieve top-ranked win rates: an 86.5% LC win rate on AlpacaEval 2.0; an 83.0 score on Arena-Hard-Auto; and a 9.55 score on MT-Bench.” A null model is maximally content-free — it ignores the prompt and returns the same canned response every time — and it tops the leaderboard. The damning part is that this defeats the defenses built to stop gaming: “several mechanisms have been developed to control length and disentangle style to reduce gameability. Nonetheless … even a ‘null model’ … can cheat.” Length control and style disentanglement are the canonical calibrate-around-the-known-bias moves, and the constant answer walks through them — the calibration-insufficiency point on the benchmark side. And it works blind, because “the instructions of these benchmarks are private and cannot be accessed”: the private grading prompt is the condition under which a transferable attack works, not a defense against one.
Push the attack down to the reward model — the scalar-valued judge that scores a response in RLHF — and the content-free property reaches its limit: the high-scoring inputs are not even language. Beyond Semantic Manipulation (arXiv 2604.02686) introduces a token-space attack optimizing “directly in token space” with “only black-box scalar feedback,” and the result is blunt: “when targeting Skywork-Reward-V2-Llama-3.1-8B, TOMPA nearly doubles the reward of GPT-5 reference answers and outperforms them on 98.0% of prompts.” A top-ranked reward model prefers the attack’s output to GPT-5’s references on 98% of prompts — and those outputs are not answers: “the generated outputs degenerate into nonsensical text, revealing that RMs can be systematically exploited beyond the semantic regime.” It is a practical RL threat, because the attack bypasses “the standard decode-re-tokenize interface between the policy and the reward model,” the exact interface an RLHF policy optimizes against — and finding high-reward token sequences is what an optimizing policy is built to do.
This holds against the strongest production reward models. Adversarial Training of Reward Models (arXiv 2504.06141) runs an automatic RL search for low-quality responses scoring highly against frontier RMs, with human-verified attack-success rates from 78.85% to 100% (“Skyworks-Gemma 27B 100; Llama-Nemotron 70B 83.01; Nemotron-340B 78.85”), and the winning responses are the content-free shapes this post keeps naming — a repeated prompt (“responses that simply repeat the prompt can achieve high scores”), inserted gibberish (“nonsensical gibberish in the middle of long text can be rewarded”), missing punctuation (“responses that lack punctuation can receive unexpectedly high scores”). The paper names the stakes: such RMs “lack robustness, awarding high rewards to low-quality, out-of-distribution (OOD) samples,” which “can lead to reward hacking, where policies exploit unintended shortcuts to maximize rewards, undermining alignment.”
There is a softer objection: fine, contrived attacks break the judge, but it works on realistic inputs, and a deploy gate sees realistic inputs. The safety-evaluation literature answers directly — the realistic adversarial inputs are where the gate fails hardest. A Coin Flip for Safety (arXiv 2603.06594) audits LLM judges used to score harmfulness in red-teaming, and finds that under the distribution shift real red-teaming induces, judge performance degrades “to near random chance,” “on average only slightly better than a random coin-flip,” against “the high human agreement reported in prior work” — grounded in “a comprehensive audit using 6642 human-verified labels,” and on deployed safety classifiers, not toy graders (“AegisGuard, the Llama-2-13B HarmBench classifier, JailJudge, and LlamaGuard-3”). And you cannot catch the manipulation in the judge’s own words: The Silent Judge (arXiv 2509.26072) changes only a content-free label on a fixed response — a date, or “Expert”/“Human”/“LLM” — and frontier verdicts move (a recency bias and “a clear provenance hierarchy (Expert > Human > LLM > Unknown),” on GPT-4o and Gemini-2.5-Flash) while “justifications almost never reference the injected cues, instead rationalizing decisions in terms of content qualities.” The judge fabricates a content-based rationale for a decision a superficial cue drove — so a justification log does not save the gate.
Three qualifications, because the version of this argument that overreaches is the one a careful reader discards.
First, content-free does not mean unfixable. The master-key paper is also a repair: data augmentation with truncated outputs as adversarial negatives produces “Master-RM,” which “achieves 0.0% FPR uniformly across all tested master keys and all five benchmarks, while keeping high standard-setting accuracy.” The 95.1% rate goes to zero. But the fix is to retrain the judge against the attack (AdvJudge-Zero similarly needs “a LoRA fine-tune stratified by a 9-class mechanism taxonomy”) — the whole point of separating this from calibration: you cannot average a master key away, you harden it. So the honest conclusion is not “LLM judges cannot be used” but “an unguarded one has on-demand false positives, and using it as a reward or gate requires hardening against the attack class.”
Second, a high false-positive rate on master keys is not the judge’s accuracy on real answers. None of these results say the judge is useless on substantive inputs — the master-key paper keeps “high performance in standard evaluation settings.” The claim is narrower: there exists a set of content-free inputs that force a false positive on demand, and the existence of that set — not the average-case accuracy — is what makes the judge unsafe as an unguarded reward or gate. An instrument forceable to a false positive by an adversary is one the adversary controls.
Third, scope. The headline rates — the 95.1%, the >90% on 22 of 24 cells, the 98.0% over GPT-5 references — are measured on specific judges, reward models, and suites, heavy on math, reasoning, and instruction-following, across the Qwen, Llama, Gemma, Skywork, and Nemotron families. The proprietary-judge result (GPT-o1, Claude-4) establishes the failure is not confined to open models, but the precise rates belong to the systems tested. Treat the numbers as evidence the attack class is broad and large, not as constants for your task.
You cannot calibrate a master key away, so the discipline is different from the bias-and-drift work a calibrated judge already needs. It is adversarial: assume the content-free attack exists, build it, and close it before the judge gates anything.
Red-team the judge with content-free inputs before you reward on it. Bare punctuation, generic reasoning openers, a repeated prompt, a constant null answer, and — if you can — token-space gibberish from a black-box search. Measure the false-positive rate each elicits, and re-run the battery on every judge-version change, because a silent vendor update can reopen the hole. A judge with a non-trivial master-key false-positive rate is not a reward signal yet, however clean its bias probes look — the same red-team-before-you-trust move an SAE feature used as a runtime monitor needs.
Assume the policy will find the hole. The attack tokens are low-perplexity strings from the judge’s own distribution — the kind an optimizing policy generates by accident and exploits on purpose. If you train against an unhardened judge, budget for reward hacking as the expected outcome, not the tail, and watch rollouts for content-free high-reward outputs.
Harden the judge against the attack class, don’t threshold around it. The fix that works is retraining on adversarial negatives — truncated outputs, mechanism-stratified examples — which drives the master-key false-positive rate toward zero. A confidence threshold or answer-order randomization does not touch a master key. If you cannot retrain, guard the inputs and treat the verdict as advisory.
Prefer comparative to absolute scoring, and don’t trust the justification. Absolute pointwise scoring is the more attackable mode; where a gate can be “is A better than B” rather than “score A,” the comparative form removes the easiest version of the attack. And the written rationale is not evidence the verdict was not gamed — the judge fabricates a content-based reason for a decision a superficial cue drove. The check is the controlled probe: same answer, vary only the superficial feature, measure the flip.
Before an LLM judge supplies a reward signal or gates a deploy:
The team measured the judge’s biases, randomized its answer order, pinned its version — every one the right thing to do to a noisy instrument. But a colon is not noise. It is a key, cut to a lock the judge did not know it had, and turning it returns correct on demand — on a frontier proprietary judge, a top-ranked reward model, a deployed safety classifier — at rates from roughly 80% to 100%. A bias you calibrate away. An on-demand false positive you have to retrain out or guard against, and an optimizing policy will find the one you left in. Probe the judge for its master keys before you reward on it, or you are training your model to find them for you.
A membership-inference attack that separates your book from a held-out one is usually reading the calendar, not the weights. A successful attack is not a sound proof — and the null it would need to test against cannot be sampled.
17 min →
Unlearning promises a model can forget specific data without a full retrain. What it delivers is output suppression: a small benign relearning set — or simply quantizing the model — brings the 'forgotten' knowledge back.
14 min →
The EU AI Act requires AI output to be marked in a robust, reliable way. A formal impossibility theorem says strong watermarking cannot exist. Five words in the statute carry the collision.
13 min →