# RL post-training launders your benchmark contamination.

A team is about to publish a reasoning model, and before the numbers go out they do the responsible thing: they run a contamination audit. The audit is the standard kind — a battery of membership-inference and memorization detectors that ask, for each benchmark, whether the model has seen its test items before. Min-K% probability, perplexity, a loss-based test, a few more. The detectors look for the fingerprint of training-set exposure: a benchmark the model trained on sits at suspiciously low perplexity, its hardest tokens assigned suspiciously high probability, because the model is partly reciting rather than solving.

The audit comes back clean. Every benchmark scores in the range a held-out, never-seen set would score. The team reads that as what it appears to be — proof the training data and the evaluation data did not overlap — and ships the model with its benchmark numbers attached, confident the numbers are earned.

The model was trained on the benchmarks. Not all of them, but enough of them, and the contamination is still in the weights — the audit did not miss it because the detectors are weak, and it did not miss it because the team ran them wrong. It missed it because the model went through a reinforcement-learning post-training stage between the contamination and the audit, and that stage did something specific and underappreciated: it erased the signal the detectors read, while leaving the contamination — and the score inflation the contamination buys — fully intact.

This post is about why an RL post-training stage defeats the contamination detectors that the field relies on, why a clean contamination audit is therefore not evidence of a clean model once RL is in the pipeline, and what an honest eval audit has to do instead.

## The mistake is trusting a detector outside its operating range

The mistake is a measurement error, and it is worth stating precisely, because it is not the obvious one. The team did not skip the audit, and they did not use bad detectors. They used good detectors outside the range where those detectors work.

Every contamination detector in standard use is a _memorization_ detector. It does not observe the training set directly — it cannot — so it infers exposure from a proxy: the model's behavior on a candidate test item, compared to its behavior on items it certainly did not see. The proxy that almost every detector uses is _likelihood_. A model that trained on a passage assigns that passage higher probability — lower perplexity, lower loss, higher Min-K% — than an equivalent passage it never saw. The whole family of likelihood-based detectors is reading one signal: the log-probability gap between members of the training set and non-members.

That signal is real, and on a model whose last training stage was supervised — pretraining, or supervised fine-tuning — the detectors work, because supervised training is _likelihood maximization_: it explicitly raises the probability of the text it trained on, which is exactly the gap the detectors measure. The detector and the training objective are reading and writing the same quantity.

Reinforcement learning is not likelihood maximization. An RL post-training stage — GRPO, PPO, the RLVR family — optimizes a _reward_, not the probability of a fixed target text, and that difference is not cosmetic. It means the signal the detectors depend on is not a signal RL is obligated to preserve. The team's error was treating "the detectors came back clean" as a model property when it is a detector property, and the detectors had been carried, by the RL stage, outside the regime where their proxy tracks the thing they are proxying for. The audit measured the memorization signature. RL had already erased it. The contamination it was a signature _of_ was still there.

## Why RL erases the signature

Two mechanisms, established in the recent literature, explain why an RL stage flattens the member/non-member gap that likelihood detectors exploit.

The first is the objective. Reward optimization does not care about the probability of any particular sequence; it cares about whether the sampled output earns reward. RLVR reinforces the model's _own_ generated responses when those responses are correct — it is not fitting a fixed target the way supervised training fits a label. So the low-perplexity dent that supervised contamination pressed into a benchmark passage is not something RL maintains. The reward gradient pushes the model toward reward-earning behavior and is indifferent to whether that behavior keeps the contaminated passage at low loss. The signature is collateral, and RL does not protect collateral.

The second is the shape of the policy after RL. RL post-training collapses the model's output distribution: the entropy falls, and the probability mass concentrates into what one of the papers below calls _highly specific and sparse modes_. A reasoning model after RL does not spread probability across many phrasings of a solution — it commits to a narrow set of high-reward trajectories. PPO's machinery sharpens this: the importance-sampling ratio between the current and old policy, combined with the clipping that bounds each update, flattens per-token log-probability differences, because updates that would push a token's probability far in either direction are clipped away. The member/non-member log-probability gap — the thing a likelihood detector measures — gets compressed along with everything else. After RL, a contaminated passage and a clean passage look about equally (im)probable to the model, not because it never saw the contaminated one, but because the policy no longer expresses its history through fine-grained likelihood.

Put the two together. RL does not optimize the signal, and RL collapses the distribution that carried the signal. A detector reading log-probabilities after an RL stage is reading a quantity that has been overwritten — and a clean reading from it is not the absence of contamination, it is the absence of the signature.

## What the research shows

The reason this is a post and not a hunch is that the laundering effect has been measured directly, on multiple detector families, and named.

[_On The Fragility of Benchmark Contamination Detection in Reasoning Models_](https://arxiv.org/abs/2510.02386) (arXiv 2510.02386) is the anchor result. It takes models deliberately contaminated by supervised fine-tuning, applies a brief GRPO stage, and re-runs the detectors. The finding is stated plainly: _even a brief GRPO training can markedly conceal contamination signals that most detection methods rely on_. Across ten detection methods and six benchmarks, detection AUROC drops by an average of 14.22 percentage points after the GRPO stage. The per-detector collapse is sharper than the average: Min-K% falls from 74.96% to 58.54%; Max-K% from 69.83% to 49.99% — 50% is the coin-flip line, so that detector is now doing nothing; the loss-based detector from 75.48% to 58.80%; LiRA, the strongest of the set, from 89.13% down to 74.89%. For the harder case — contamination introduced at the final stage as chain-of-thought traces on an already-advanced reasoning model — most detectors collapse to near 50%, random. And the contamination is still working: the SFT contamination inflates pass@1 by 8.82%, and the final-stage CoT contamination inflates it by 11.76%, while, in the paper's words, _leaving little evidence_. The score went up by eight to twelve points; the audit shows nothing.

[_Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models_](https://arxiv.org/abs/2510.09259) (arXiv 2510.09259) isolates the case where the contamination enters during the RL stage itself, and the headline is blunt: _existing methods are close to a random guess for RL-phase contamination_. The likelihood-based detectors — Perplexity, Min-K%, Min-K++ — average roughly 0.44 to 0.51 AUC against RL-phase contamination, which is to say they are not detecting it at all. The paper gives the cause in the terms above: after RL the output entropy collapses into highly specific and sparse modes, and RL decouples from the likelihood objective, erasing the low-perplexity signal the detectors exploit. The authors propose a detector built for the new regime — Self-Critique, which probes the model's behavior rather than its likelihoods — and report it lifts detection to roughly 0.70 average AUC, an average gain of about 19% over the likelihood baselines; the abstract states the improvement as _up to 30%_, which is the maximum across settings, not the typical one. They also release RL-MIA, a benchmark for membership inference under RL post-training.

[_Auditing Data Membership in Reinforcement Learning With Verifiable Rewards_](https://arxiv.org/abs/2511.14045) (arXiv 2511.14045) confirms the failure for RLVR specifically and explains it from the membership-inference side: _traditional membership inference attacks fail_ on RLVR because it reinforces model-generated responses rather than fitting fixed targets, so the attack has no memorized target to detect. The numbers are stark — on DAPO-7B, calibrated Min-K% reaches 0.517 AUC and calibrated Loss 0.571 AUC, both near chance, against the paper's proposed DIBA detector at 0.819. At the operating point that matters for an audit, true-positive rate at 0.1% false-positive rate, the likelihood baselines run at roughly 0.003 to 0.005 — effectively zero, they catch nothing — while DIBA reaches 0.073 to 0.107.

One more result names the second, worse half of the problem. [_The Impact of Post-training on Data Contamination_](https://arxiv.org/abs/2601.06103) (arXiv 2601.06103) finds that GRPO does not only inflate scores on the benchmark that was leaked — it inflates scores on _uncontaminated_ benchmarks too. Contaminate GSM8K and run GRPO, and the measurable gains show up on GSMPlus and HumanEval, benchmarks that were never in the contaminated data. SFT contamination, by contrast, inflates only the contaminated task. The paper's framing of the mechanism is the one to keep: the leaked information is _merely submerged, not erased_ — which is exactly the careful claim this post is making. RL does not remove the contamination. It hides it, and the hidden contamination still pays out, even onto benchmarks you never touched.

## What the claim does not cover

Four qualifications, because the precise version of this argument is the only one worth making.

The first is the most important, and it is about a single word. RL does not _remove_ contamination — it defeats the detectors. The contamination persists in the weights; the proof that it persists is the score inflation, which does not go away when the detector signal does. "Launders" is the right verb because laundering hides the origin of something that is still there; "erases" would be the wrong verb if it were taken to mean the contamination is gone. Throughout, the claim is that RL erases the _detectable signature_, not the contamination.

Second, the "inflates never-leaked benchmarks" finding rests mainly on one paper, 2601.06103. It is a real and measured result, but it is more narrowly evidenced than the core laundering claim, which is supported by three independent papers — treat the cross-benchmark inflation as established-but-newer rather than as settled as the detector-collapse result.

Third, the scope. Every result here is on reasoning models, on math and code benchmarks, at model scales of roughly 7 to 14 billion parameters, with GRPO- and PPO-family RL. The mechanism — reward optimization plus entropy collapse erasing a likelihood signal — should generalize, but it has been measured in that box, and the numbers belong to it. The "up to 30%" figure for the Self-Critique detector is a maximum across settings; the typical gain is about 19%.

Fourth, this is not a counsel of despair, and the papers do not read as one. The same papers that demonstrate the detector collapse also propose detectors built for the RL regime — Self-Critique, DIBA — that restore meaningful detection. Contamination auditing under RL post-training is an arms race, not a dead end. The honest conclusion is not "auditing is hopeless"; it is "the standard likelihood-based audit is insufficient once RL is in the pipeline, and you need the RL-aware tools."

## Auditing a model that has been through RL

You cannot make a likelihood detector work on a model whose policy no longer expresses its history through likelihood. What you can do is stop treating a clean standard audit as a clean bill of health, and build an audit that matches the pipeline.

**Treat a clean contamination audit as inconclusive, not negative, for any RL-post-trained model.** This is the decision everything else follows from. If the model went through GRPO, PPO, or any RLVR stage, a passing result from Min-K%, perplexity, or loss-based detection is consistent with a clean model _and_ with a thoroughly contaminated one whose signature was laundered. The audit has not cleared the model; it has failed to say anything. Record it as inconclusive.

**Use RL-aware detectors, not only likelihood-based ones.** The likelihood family — Min-K%, Min-K++, perplexity, loss — runs at chance on RL-phase contamination. Detectors that probe behavior rather than log-probabilities — the Self-Critique and DIBA approaches — recover real signal: roughly 0.70 AUC and 0.819 AUC respectively where the likelihood baselines sit near 0.50. An audit pipeline for a reasoning model needs at least one detector from the RL-aware family, and it should be weighted above the likelihood detectors, not averaged in as an equal.

**Audit before the RL stage, not only after.** The cleanest moment to detect supervised contamination is before RL has had a chance to launder it. If you control the training pipeline, run the contamination audit on the pre-RL checkpoint — the SFT or pretrained model — where likelihood detectors still work, and keep that result. A model that is clean before RL and clean after RL is a meaningfully stronger claim than a model that is only clean after.

**Cross-check for inflation on benchmarks you did not train on.** Because GRPO inflation leaks onto uncontaminated benchmarks, a suspiciously strong score on a benchmark you are sure was never in the data is not reassurance — it can be the spillover of contamination on a _different_ benchmark. Compare the model's profile across related benchmarks; a model that is strong everywhere in a family including the held-out members is a model to look at harder, not to trust more.

**Weigh capability evidence, not just signature evidence.** The contamination persists as capability — that is what the +8-to-12% pass@1 inflation is. So an audit should include capability-side probes: held-out benchmarks the model has provably never seen, freshly authored items, and per-difficulty breakdowns, in the spirit of the dynamic, continuously-refreshed evaluation that [a maintained golden set](/blog/golden-set-maintenance/) and [eval-driven development](/blog/eval-driven-development/) already argue for. A signature-only audit asks "can I see the fingerprint"; a capability-aware audit also asks "is the score plausible for a model that never saw this."

**Track provenance, because a detector you cannot trust makes provenance the audit.** When the post-hoc detector is unreliable, the trustworthy record is the one made at training time: what data went into which stage, attested and logged. This is the same argument [model weights need provenance](/blog/weights-provenance/) makes for the supply chain — if you cannot reliably detect contamination after the fact, the defensible position is a documented training-data trail, not a clean detector reading.

## The checklist

Before you trust a contamination audit on an RL-post-trained model:

- [ ] Whether the model went through an RL stage (GRPO, PPO, RLVR) is known and recorded, because it changes how every detector result is read.
- [ ] A clean result from likelihood-based detectors — Min-K%, perplexity, loss — on an RL-post-trained model is logged as inconclusive, not as negative.
- [ ] At least one RL-aware detector (Self-Critique- or DIBA-style, probing behavior rather than likelihood) is in the audit, and weighted above the likelihood detectors.
- [ ] Where the training pipeline is controlled, the contamination audit is also run on the pre-RL checkpoint and that result is kept.
- [ ] Scores are cross-checked across benchmark families for inflation spilling onto benchmarks that were never in the training data.
- [ ] The audit includes capability-side probes — freshly authored, provably-unseen items — not only memorization-signature detectors.
- [ ] A training-data provenance record exists, so a contaminated model is not cleared solely by an unreliable post-hoc detector.
- [ ] Someone has accepted that contamination auditing under RL is an arms race, and the audit is scheduled to be re-qualified as detectors improve.

## Reading list

- [_On The Fragility of Benchmark Contamination Detection in Reasoning Models_](https://arxiv.org/abs/2510.02386) (arXiv 2510.02386) — a brief GRPO stage drops detection AUROC by an average of 14.22 points across ten methods and six benchmarks (Max-K% collapses to ~50%, random), while SFT contamination still inflates pass@1 by 8.82% and final-stage CoT contamination by 11.76%.
- [_Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models_](https://arxiv.org/abs/2510.09259) (arXiv 2510.09259) — likelihood detectors run at ~0.44-0.51 AUC on RL-phase contamination because post-RL entropy collapses into sparse modes; the proposed Self-Critique detector reaches ~0.70 AUC (an ~19% average gain, up to 30%), and introduces the RL-MIA benchmark.
- [_Auditing Data Membership in Reinforcement Learning With Verifiable Rewards_](https://arxiv.org/abs/2511.14045) (arXiv 2511.14045) — traditional membership inference fails on RLVR because it reinforces model-generated responses; likelihood baselines run near chance (DAPO-7B Min-K% 0.517 AUC) and at ~0 true-positive rate at 0.1% false-positive rate, while the proposed DIBA reaches 0.819 AUC.
- [_The Impact of Post-training on Data Contamination_](https://arxiv.org/abs/2601.06103) (arXiv 2601.06103) — GRPO inflates scores on uncontaminated benchmarks (GSM8K contamination lifts GSMPlus and HumanEval) while SFT inflates only contaminated tasks, and the leaked information is "merely submerged, not erased."

The contamination audit came back clean, and the team read it the way the word "clean" invites — as the model and the eval data never having met. But the audit they ran measures a memorization fingerprint, and the RL stage in their pipeline had wiped the fingerprint off a model whose hands were still dirty. A passing likelihood detector on an RL-post-trained model is not an acquittal. It is a detector reporting, accurately, that it can no longer see — and the score inflation, sitting right there in the benchmark numbers, is the evidence it can no longer find.