GRPO's group-relative advantage looks like a clean normalized policy gradient. Two of its normalizers aren't neutral: dividing by response length rewards longer wrong answers, dividing by reward std overweights easy and hard questions. The trainer optimizes both, faithfully.
A team runs GRPO on a math-reasoning model and watches the curves do the thing the field has taught them to want. Reward climbs. Accuracy ticks up. And the average response length climbs too — steadily, run after run, the model writing longer and longer derivations. Someone calls it out in the standup as the good sign everyone has been primed to see: the model is learning to think for longer before it answers. The release notes will say the RL stage produced emergent reasoning, and the evidence offered will be the length curve — the model now spends more tokens on a problem than it did before training, read automatically as the model working harder.
Part of that length curve is real reasoning. And part of it is the trainer faithfully optimizing a bias baked into the advantage estimator — one the team did not choose, did not notice, and is now reporting as a capability.
GRPO’s advantage looks like the cleanest object in the recipe. You sample a group of responses to a prompt, score each one, subtract the group mean, divide by the group standard deviation, and you have a normalized, baseline-free, low-variance policy-gradient signal — no value network, no critic, just the group telling you which of its members did better than average. It is the part of the method that feels least like a knob and most like math. But two of the operations in that clean definition are not neutral. The division by response length and the division by the group’s reward standard deviation each inject a systematic, predictable distortion into the objective — and the trainer, which optimizes exactly the objective you hand it, optimizes those distortions too.
This post is about the two biases hiding in the GRPO advantage estimator: where each comes from in the arithmetic, why the response-length inflation that gets read as emergent reasoning is partly an artifact of the first, and why the fixes live at the estimator — not the task, not the learning rate. It also covers the honest tension, because a strong result argues the std normalizer is doing useful work, and “just delete it” is not unconditionally free.
Start with the first bias, because it is the one wearing the costume of a capability.
GRPO assigns one scalar advantage to a whole response, then spreads it across the response’s tokens — and the spreading divides by the response length, the 1/|o_i| term. The intent is innocuous: a long response and a short one should be comparable, so you average the per-token contribution rather than letting longer responses dominate by token count alone. But look at what that division does when the advantage is negative — when the response was wrong and the policy is being pushed away from it. The penalty is divided by length. A longer wrong answer absorbs the same total penalty spread over more tokens, so each token is penalized less, and the gradient pushes less hard against a long wrong answer than a short one.
Understanding R1-Zero-Like Training: A Critical Perspective (arXiv 2503.20783) states the mechanism exactly: “for negative advantages (A^i,t < 0, indicating an incorrect response), longer responses are penalized less due to their larger |o_i|, causing the policy to prefer lengthier responses among incorrect ones.” Read that as an optimizer would. Among the responses the model gets wrong, the estimator has a built-in preference for the long ones — not because long wrong answers are better, but because the normalizer charges them a smaller per-token penalty. The policy climbs that gradient like any other, wrong answers get longer over training, and because wrong answers are a large share of what a model on a hard benchmark produces, the average length inflates. The paper names this directly: “an optimization bias in Group Relative Policy Optimization (GRPO), which artificially increases response length (especially for incorrect outputs) during training.”
That should change how the length curve reads. A rising average-length curve is consistent with the model learning to reason for longer, and also consistent with the estimator quietly preferring lengthier incorrect outputs — and the two are not distinguishable from the curve alone. The fraction of length growth that sits in responses still scored wrong is not emergent reasoning; it is the 1/|o_i| term doing what its arithmetic forces.
The same paper proposes the obvious correction: remove the offending normalizer. “We propose to simply remove the 1/|o_i| and std normalization terms … Dr. GRPO can effectively mitigate the optimization bias and lead to better token efficiency.” It is a one-line change to how the advantage is aggregated, at the level of the estimator — not the task, the reward, or the schedule. And it does not cost accuracy: the minimalist Dr. GRPO recipe reached 43.3% accuracy on AIME 2024 with a 7B base model, and Oat-Zero-7B reached an average of 51.4% across five datasets (AIME24, AMC, MATH500, Minerva, OlympiadBench). Killing the length bias did not kill the reasoning. It killed the inflation that was masquerading as reasoning.
The reason this is a post and not one paper’s idiosyncratic complaint is that an independent team, building a different system, diagnosed the same length artifact at a different point in the math — the loss aggregation rather than the per-response advantage — and reached for the same kind of fix.
DAPO: An Open-Source LLM Reinforcement Learning System at Scale (arXiv 2503.14476) looks at how GRPO combines losses across a batch: it normalizes within each sample first, then averages across samples, so every sample gets the same weight regardless of its token count. The consequence is the same dilution from the other direction: “Since all samples are assigned the same weight in the loss calculation, tokens within longer responses (which contain more tokens) may have a disproportionately lower contribution to the overall loss.” A token inside a long response moves the gradient less than one inside a short response, purely because of where it sits.
DAPO connects that choice to the same pathology Dr. GRPO flagged: “sample-level loss calculation, due to its inability to effectively penalize those undesirable patterns in long samples, leads to an unhealthy increase in entropy and response length.” Two teams, two entry points into the GRPO objective, one symptom. DAPO’s correction is its Token-Level Policy Gradient Loss, which replaces GRPO’s “sample-level loss calculation, which involves first averaging the losses by token within each sample and then aggregating the losses across samples” with a per-token aggregation, so that “longer sequences can have more influence on the overall gradient update compared to shorter sequences.” Make every token count proportionally and the long-response discount disappears — and DAPO still reaches 50 points on AIME 2024 with Qwen2.5-32B.
A third paper frames the length bias in its most general form, as a property of the advantage structure rather than any one aggregation choice. lambda-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferences (arXiv 2510.06870) puts it cleanly: “GRPO inherently suffers from a length bias, since the same advantage is uniformly assigned to all tokens of a response.” One scalar advantage spread across however many tokens a response has means the per-token weight is implicitly a function of length, so “longer responses distribute the reward over more tokens and thus contribute disproportionately to gradient updates.” The word that matters is inherently: this is not a tuning accident you can dial out with a learning rate, it is in the shape of the estimator. lambda-GRPO also characterizes the existing fixes — DAPO, Dr. GRPO — as “heuristic” token-aggregation schemes, and its own learnable-weight alternative improves over vanilla GRPO by a modest “+1.9% (1.5B), +1.0% (3B), and +1.7% (7B) average accuracy, without any modifications to the training data or additional computational cost.” Single-digit gains — the honest size of the effect once you are past the gross bias, and a calibration against expecting the fix to transform a model.
The second bias lives in the other normalizer — the division by the group’s reward standard deviation — and it distorts a different axis: not length, but which questions drive the update.
Picture two prompts. One sits at the model’s frontier: some sampled responses are correct, some wrong, so the rewards have real spread and the standard deviation is healthy. The other is too easy or too hard — the model gets it right almost every time, or wrong almost every time, so the group rewards are nearly all 1 or all 0 and the standard deviation is tiny. Divide each advantage by its standard deviation, and the lopsided prompt is divided by a near-zero number: its contribution to the update blows up, while the frontier prompt gets a moderate weight.
Understanding R1-Zero-Like Training (arXiv 2503.20783) names this too: “Questions with lower standard deviations (e.g., those that are too easy or too hard, with the outcome rewards being almost all 1 or 0) are given higher weights during policy updates.” That is backwards. The frontier prompts — where the model is genuinely on the boundary between can and cannot, where the signal is richest — get down-weighted relative to the ones it has already mastered or has no purchase on. The estimator spends its gradient budget on the prompts that teach the least.
A separate paper makes a closely related claim its title. Your Group-Relative Advantage Is Biased (arXiv 2601.08521) shows the group-relative advantage estimator is biased relative to the true advantage, “systematically underestimating advantages for hard prompts and overestimating them for easy prompts.” Its mechanism is the group baseline rather than the std term specifically — excluding degenerate all-correct or all-wrong groups makes the empirical mean a biased estimate, which skews the advantage by difficulty — so it is a different route to the same difficulty-dependent distortion the std normalizer produces, not a second derivation of the std effect. Its fix, again, lives at the estimator level: “History-Aware Adaptive Difficulty Weighting (HA-DW), which dynamically adjusts advantage weights based on the model’s evolving state while incorporating long-term reward signals” — a reweighting of the advantage, not a change to the prompts or the reward.
The std problem is not only about easy-versus-hard. Stratified GRPO: Handling Structural Heterogeneity in Reinforcement Learning of LLM Search Agents (arXiv 2510.06214) identifies “cross-stratum bias - an ‘apples-to-oranges’ comparison of heterogeneous trajectories” that arises whenever a single group baseline and std are computed over structurally unlike items, which “distorts credit assignment and hinders exploration.” Its corrected normalizer partitions trajectories into homogeneous strata and normalizes within each, which the paper proves “eliminates cross-stratum bias” while retaining “the global unbiasedness and unit-variance properties enjoyed by standard normalization.” That the corrected normalizer “outperforms GRPO by up to 11.3 points” says the mis-scaling is not academic: where the group is heterogeneous, fixing the scaling is worth double digits.
Here the literature pushes back on itself, and the pushback has to be presented as such, because the tidy conclusion — “the normalizers are bugs, delete them” — runs into a serious paper arguing the opposite.
Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradients (arXiv 2601.23135) argues that dividing by the reward standard deviation is not merely a bias-injecting nuisance — it doubles as an adaptive step-size rule: “Standard-deviation normalization therefore implements an adaptive gradient rule, scaling each update by an inverse-curvature proxy; the reward variance of each question serves as an estimate of the local Lipschitz constant (curvature) of the policy gradient.” The same std term that overweights low-variance prompts is, through a curvature lens, scaling each prompt’s update by something matched to its local smoothness — exactly what an adaptive optimizer should do. The “bias” and the “feature” are the same arithmetic from two angles.
And the trade-off has teeth. Removing the normalizer is not free in their experiments: across the intermediate Phase II (iterations 100-300), the run with normalization opens a gap over the one without. On GSM8K-Easy, “standard reaches ≈0.95 while no_std attains ≈0.92, opening a gap of nearly 3 percentage points”; on GSM8K-Hard the separation is larger — “standard climbs to ≈0.85 while no_std reaches only ≈0.78, yielding a ∼7 percentage point difference at iteration 300.” The std term is buying convergence, and deleting it gives some back. So the honest framing is not “normalization is bad” but that the std normalizer trades a known difficulty bias for a convergence benefit, and which side you want depends on your regime. Dr. GRPO removed it without losing accuracy in a minimalist R1-Zero recipe; this paper shows a setting where removing it costs several points. Both are real — the normalizers are a design choice with a genuine tension underneath, not a settled bug.
A second qualification, about the length half of the thesis: the claim is not that all response-length growth is an artifact, only that part of it is and gets misattributed. The negative-gradient channel Dr. GRPO flags as the source of inflation is the same one that can be engineered into useful exploration. e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs (arXiv 2506.09026) leans on it on purpose: “leveraging ‘negative’ gradients from incorrect traces to amplify exploration during RL, resulting in longer search traces that chains additional asymmetries.” The difference from the naive read is that e3 engineers the longer traces, “coupling task difficulty with training token budget during training via a specifically-designed curriculum” — they are not a free byproduct that certifies itself as reasoning. Whether longer is better is a property of the recipe you built. The same caution applies that RL sharpens reasoning, it doesn’t expand it raises: a headline RL metric can move for reasons other than the capability you think you bought, and the length curve is one more such number.
State the takeaways in the form that helps a team, scoped to what the sources establish.
Decompose the length curve before you call it reasoning. A rising average-length curve is not, by itself, evidence of emergent reasoning — under vanilla GRPO it is partly the 1/|o_i| term preferring lengthier incorrect outputs. Split the growth by correctness. Growth concentrated in responses that are still wrong is the estimator artifact, not a capability, and reporting it as reasoning is reporting a bug as a feature.
Treat the two normalizers as choices, not math you inherit. Dr. GRPO removes both and keeps accuracy; DAPO replaces the length-diluting aggregation with a token-level loss; lambda-GRPO learns the token weighting. Which fix fits is empirical, but the decision is yours to make deliberately — including the choice to keep the std term where its adaptive-step-size role earns its keep.
For the difficulty bias, reweight at the estimator, not the dataset. The std term over-weights questions the model has already mastered or cannot touch and starves the frontier where the signal is richest. The literature’s remedies — difficulty-aware reweighting, stratified normalization — adjust the advantage estimate; they do not change which prompts you train on. Stratified GRPO’s up-to-11.3-point gain says the mis-scaling is worth correcting where the group is heterogeneous.
Two qualifications, because the overreaching version of this argument is the one a reader who knows the field stops trusting.
First, the evidence is concentrated on math, code, and search-agent reasoning, on models roughly 32B and below, trained with R1-Zero-style verifiable-reward recipes. The mechanisms — a length normalizer that discounts long wrong answers, a std normalizer that explodes on low-variance groups — are properties of the arithmetic and should generalize, but the magnitudes are from this box and should not be quoted outside it without evidence.
Second, the biases are real but their sizes vary, and some corrections buy single-digit gains, not transformations — lambda-GRPO’s +1.0% to +1.9% is the honest order of magnitude once the gross bias is handled, while the double-digit result (Stratified GRPO’s up to 11.3 points) lives in a setting of strong structural heterogeneity. Do not assume every GRPO run is leaving eleven points on the table; assume it has a bias whose cost is worth measuring. “Biased estimator” is a precise claim, not a moral one — a biased estimator can still be the right tool, and the fix is to make the choice consciously, which sometimes means keeping the normalizer.
Before you read a GRPO length curve as emergent reasoning, or sign off on the advantage estimator as a neutral baseline:
1/|o_i| length normalizer is a deliberate choice — kept, removed (Dr. GRPO), or replaced with a token-level loss (DAPO) — not an unexamined default.The length curve climbed and the team read it as the model learning to think for longer. The curve was real; the reading skipped a step. Under vanilla GRPO, part of that growth is the estimator preferring longer wrong answers, and part of the gradient is spent on the easy and hard questions the std normalizer inflates — neither chosen by anyone, both optimized because they sit inside the objective. Decompose the length curve by correctness, treat the two normalizers as decisions instead of arithmetic, and you will know which part of the run was reasoning and which part was the estimator. The model may well be reasoning more. Just don’t let the bias take the credit.
A big pass@1 jump after a GRPO run reads as the model having learned to reason better. Often it has not — it has learned to pick, more reliably, a solution path it already had. RLVR re-weights the base model's existing reasoning rather than creating new reasoning, and whether any RL budget escapes that is genuinely disputed.
14 min →
Chinchilla's twenty-to-one was fit on unique data, one pass. Teams now repeat scarce corpora at tens of times that ratio, where the law breaks — and past a repetition threshold the re-derivations reverse its verdict: spend compute on capacity, not more epochs.
14 min →
Four-bit quantization looks like a deployment-time decision: pick a method, run it, recover the accuracy. It is not — genuine low-bit quantizability is fixed at pre-training time by optimizer and normalization choices, and the post-hoc rotation tricks you reach for at deploy time yield less the lower the bit-width goes.
15 min →