A big pass@1 jump after a GRPO run reads as the model having learned to reason better. Often it has not — it has learned to pick, more reliably, a solution path it already had. RLVR re-weights the base model's existing reasoning rather than creating new reasoning, and whether any RL budget escapes that is genuinely disputed.
A team has a capable base model and wants it to reason better. They reach for the method the field has converged on: reinforcement learning with verifiable rewards — GRPO, or one of its siblings — where the model generates solutions, a checker scores them right or wrong, and the policy is nudged toward the ones that pass. They run it on a math-reasoning set, and the headline metric moves the way the recipe promises. Pass@1 on the held-out benchmark jumps — ten points, fifteen, more on the harder sets. The model answers more questions correctly on its first try than it did before.
The interpretation writes itself, and a design review would accept it without a pause: the RL stage taught the model to reason. The pass@1 jump is large, it is on held-out problems, and it followed directly from the training — every signal a team uses to decide that a method worked is present. The model is then described, in the release notes and the internal docs, as having expanded reasoning capability. It can do something it could not do before.
That description is probably wrong, and the gap between it and the truth is not rhetorical — it is measurable, and it has been measured. Pass@1 is one number from a distribution, and it is the number RLVR moves most and most misleadingly. Sample the same model many times per problem instead of once — pass@k, for k in the hundreds — and a different picture appears: the base model, given enough attempts, often solves as many distinct problems as the RL model, sometimes more. The RL stage did not add reasoning paths the base model lacked. It raised the probability of the paths the base model already had.
This post is about why RLVR predominantly re-weights existing reasoning rather than creating new reasoning, why that distinction is invisible to pass@1 and stark under pass@k, and what — if anything — actually expands a model’s reasoning capability.
The mistake is a measurement error, and it is worth stating precisely, because pass@1 is such a natural thing to optimize and report.
Pass@1 and pass@k measure two different properties, and the difference is the whole argument. Pass@1 is the probability that a single sample from the model solves the problem — it is a measure of the model’s propensity, how reliably it reaches a correct answer on one try. Pass@k is the probability that at least one of k samples solves it — it is a measure of the model’s coverage, whether a correct solution path exists anywhere in the model’s distribution at all, even at low probability. Propensity is about reliability. Coverage is about capability — the set of problems the model can solve, given enough attempts.
RLVR is, by construction, a propensity optimizer. The reward signal is “this sampled solution was correct,” and the policy update raises the probability of correct-scoring trajectories and lowers the rest. That sharpens the distribution toward solution paths that already earn reward — which means it acts on paths the model can already produce, because a path the model never samples never earns reward and never gets reinforced. The mechanism has no handle on coverage. It cannot reward a solution the model does not yet generate, so it cannot, by its own dynamics, add one.
So “pass@1 jumped fifteen points” is a true statement about propensity that gets read as a statement about coverage. The model became more reliable at reaching answers it could already, with some probability, reach. Whether the set of reachable answers grew is a different question, pass@1 does not ask it, and the recipe’s own mechanics suggest the answer is no.
The reason this is a post and not a hunch is that the propensity-versus-coverage gap has been measured directly, by sweeping k.
Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (arXiv 2504.13837) is the anchor. It evaluates base models and their RLVR-trained counterparts not at pass@1 but across pass@k, with k pushed into the hundreds and beyond, and the curves cross. On MATH500, the base model overtakes the RL model at around k=128 — below that the RL model leads, above it the base model does — and on AIME24 the gap widens through k=256 to 1024 in the base model’s favor. The concrete figures, read off the curves and so approximate: Qwen2.5-7B on AIME24 trained with SimpleRL shows RL pass@1 around 26% against the base model’s roughly 10% — the headline RL win — but at pass@256 the base model scores about 69% against the RL model’s about 68%. The propensity gain is real and large; the coverage gain is gone.
The sharpest evidence in that paper is the solution-coverage breakdown. On AIME24, of the problems solved, 63.3% were solved by both the base and the RL model, 13.3% were solved by the base model only — and 0.0% were solved by the RL model only. Zero. Across that benchmark the RL stage discovered no problem the base model could not already solve given enough samples. That is the propensity-optimizer mechanism showing up in the data exactly as predicted: RL moved probability around inside the base model’s existing coverage and added nothing outside it. The same paper notes that a distilled model — DeepSeek-R1-Distill-Qwen-7B — shows pass@k curves sitting above both the base and the RL model, which is the first hint at what does expand coverage.
Reinforcement Learning vs. Distillation: Understanding Accuracy and Capability in LLM Reasoning (arXiv 2505.14216) puts the same split on a single clean pair of numbers. On Qwen2.5-1.5B-Math evaluated on MATH500, RLVR raises accuracy by 13.9 percentage points — 60.6% to 74.2% — while pass@256 capability moves from 97.2% to 97.0%, a change of -0.2 points, flat within noise. Accuracy up fourteen points, capability unchanged: the two metrics, measured on one model on one benchmark, move independently, and that independence is the thesis. The paper also locates where the propensity gain lands: RLVR’s accuracy gains concentrate on mid-difficulty questions — a +36.6% gain on the medium [65-128] difficulty bin — and contribute essentially nothing on the hardest bin, +0.5%. RL buys reliability on problems of middling hardness; it abandons the hardest problems, because a problem the model almost never solves almost never produces a correct sample to reinforce.
If pass@k is the symptom, the mechanism is what one paper calls the invisible leash: RLVR is mathematically bounded by the support of the base model’s distribution.
The Invisible Leash: Why RLVR May or May Not Escape Its Origin (arXiv 2507.14843) makes the support-boundedness precise and measures it. Three quantities carry the result. The Support Retention Rate — the fraction of the base model’s solution support that survives RL — runs about 0.93 to 0.99: RL keeps almost everything the base model had. The Net Discovery Rate — genuinely new solutions RL adds outside the base support — is at most 0.04: RL adds almost nothing. And the Net Support Change Rate is negative: on balance the support shrinks. RLVR is not a slightly-leaky boundary expander. It is a within-support reweighting that, if anything, contracts the set of things the model can produce, because every probability it concentrates onto reward-earning trajectories is probability it removes from the long tail — and the long tail is where rare, hard-won solutions live.
That contraction shows up as a diversity collapse, and the paper measures it on the model the dispute is about. ProRL — a prolonged-RL training run, discussed in the next section — improves avg@32 from 54.5% to 65.4%, a real propensity gain. But over the same training its answer-level entropy drops from 1.30 to 0.66: the model’s distribution over answers collapses to roughly half its prior spread. The paper’s blunt accounting: ProRL-1.5B loses 175 completions while gaining only 48. It gains correct answers on some problems and loses the ability to produce a correct answer on more — a net reduction in coverage dressed, by pass@1, as an improvement.
Here the literature genuinely splits, and the split has to be presented as a split, because one strong paper argues the opposite of everything above and it deserves a fair hearing rather than a dismissal.
ProRL (arXiv 2505.24864) argues that the support-boundedness result is an artifact of insufficient RL, not a law of RL. Its claim is that prolonged RL training — far more steps, with the training stabilized so it does not collapse early — does expand the reasoning boundary: ProRL reports that its models solve problems the base model fails even when the base model is given a large sampling budget, which is precisely the coverage gain the papers above say RLVR cannot produce. If that holds, the honest framing is not “RL sharpens, it cannot expand” but “short RL sharpens, long RL can expand” — and a team with a real compute budget would draw a different conclusion. ProRL is not a strawman. It is a serious result with a serious mechanism — that the boundary looks fixed only because nobody trained long enough to move it — and it is the live counter-position to this post’s thesis.
The rebuttal is direct, and it comes from the Invisible Leash paper, which engages ProRL by name. Its measurements are on ProRL’s own models, and they land inside ProRL’s claim. ProRL’s avg@32 gain is real, but it co-occurs with the answer-entropy collapse from 1.30 to 0.66 and the net loss of 175 completions for 48 gained — the gain is a reweighting, not an expansion. And the coverage test is decisive: at very large k, base model AIME2024 pass@8192 = 93.3% against ProRL-1.5B’s 83.3%. Given enough attempts, the base model solves more of AIME2024 than the prolonged-RL model does. ProRL’s gains, on this measurement, sit inside the base model’s support after all — prolonged RL sharpened harder and contracted the tail harder, but it did not escape the origin. The question is not closed by this exchange: ProRL stands as a real claim, the Invisible Leash rebuts it on ProRL’s own checkpoints, and a careful practitioner treats “can a larger RL budget expand coverage” as actively disputed — not as settled in either direction.
Three qualifications, because the precise version of this argument is the only one that survives contact with someone who knows the field.
First, all the on-thesis evidence is math, code, and STEM reasoning, on models of roughly 32B parameters and below. The propensity-versus-coverage split is well established in that box. It is not measured at frontier scale, and it is not measured on open-ended non-verifiable domains — do not generalize it there without evidence.
Second, “RLVR lowers pass@k at large k” is a statement about large k. At small k — k=1 through, roughly, the crossover near k=128 in the Does-RL-Really paper — the RL model leads, because higher propensity is exactly what wins when you only get a few samples. The crossover is the whole shape: RL helps in the few-sample regime and hurts in the many-sample regime. If your deployment takes one sample per query, the propensity gain is genuinely the gain you want.
Third, the contested question is contested. ProRL is a live counter-position with a serious mechanism, and this post presents it as one. The robust, uncontested claim is the narrow one: a short GRPO pass re-weights existing reasoning and does not add coverage. Whether a long, well-tuned RL run can do more is disputed, and the honest stance is to say so rather than to declare the boundary fixed.
You cannot make a propensity optimizer add coverage; rewarding only sampled-and-correct trajectories is what RLVR is. What you can do is match the method to the goal, and measure the property you actually changed.
Use GRPO to sharpen a model that is already capable, not to create a capability. RLVR’s reliable, uncontested effect is to raise pass@1 by concentrating probability on solution paths the base model already has. That is genuinely valuable when the base model can already reach the answer and you need it to do so reliably on the first try. It is the right tool for converting latent capability into reliable capability — and the wrong tool for installing a capability that is not latent.
Measure pass@k across a sweep, not pass@1 alone. A pass@1 jump with a flat or falling pass@k curve is a propensity gain, not a capability gain, and only the sweep tells you which you bought. Report the curve and note where it crosses the base model — that crossover is the honest summary of what the RL run did.
Watch answer-level diversity as a cost, not a side effect. RLVR contracts the output distribution; entropy falls, the tail thins. Track answer entropy or solution diversity across training and treat its collapse as a real cost — because a model that has lost its rare-but-correct completions has lost coverage, whatever pass@1 says.
To add genuinely new reasoning capability, distill with new knowledge. The route that expands coverage in the literature is distillation that introduces knowledge the student did not have — the DeepSeek distill in the RL-vs-distillation paper reaches AIME25 pass@256 of 70.0% against the base model’s 56.7%, a real coverage gain. The crucial qualifier: distillation expands capability only when it introduces new knowledge; the same paper finds that distilling reasoning patterns alone, with no new knowledge, behaves like RLVR — a propensity gain inside the existing support. Distillation is not magic; introducing new knowledge is the active ingredient.
Do not assume a longer RL run settles the question. ProRL argues prolonged RL expands the boundary; the Invisible Leash rebuts it on ProRL’s own checkpoints. If you have the budget to run long RL, run it as an experiment with a pass@k sweep and an entropy track — not as a settled recipe — and let your own coverage measurement, not the recipe’s reputation, tell you what happened.
This is the same discipline that keeping the eval through distillation and choosing fine-tuning over prompting deliberately already argue for, and it pairs with the caution in process reward models reward fluency: a reward signal optimizes the thing it measures, and pass@1 is not capability. The data the RL stage runs on matters as much as the method — RL environments are the new dataset — but no environment changes the fact that a propensity optimizer reweights; it does not invent.
Before you describe an RLVR run as having expanded a model’s reasoning:
The pass@1 jump after the GRPO run was real, and the team that read it as the model learning to reason was reading a real number. Their error was upstream of it: they read a propensity metric as a capability metric, and the two come apart exactly where RLVR is concerned. The RL stage made the model more reliable at the reasoning it already had — a genuine and useful thing — and it did not, on the measurements, hand the model reasoning it lacked. Sweep pass@k against the base model before you write “expanded.” If the curves cross, what you sharpened was a knife you already owned.
GRPO's group-relative advantage looks like a clean normalized policy gradient. Two of its normalizers aren't neutral: dividing by response length rewards longer wrong answers, dividing by reward std overweights easy and hard questions. The trainer optimizes both, faithfully.
16 min →
Chinchilla's twenty-to-one was fit on unique data, one pass. Teams now repeat scarce corpora at tens of times that ratio, where the law breaks — and past a repetition threshold the re-derivations reverse its verdict: spend compute on capacity, not more epochs.
14 min →
Four-bit quantization looks like a deployment-time decision: pick a method, run it, recover the accuracy. It is not — genuine low-bit quantizability is fixed at pre-training time by optimizer and normalization choices, and the post-hoc rotation tricks you reach for at deploy time yield less the lower the bit-width goes.
15 min →