# The same RLHF that made your model helpful made its confidence a liar.

A team is building a high-stakes assistant — medical triage, legal intake, something where a confidently wrong answer is worse than no answer — and they make the responsible design choice: the model should know when it does not know. So they wire in confidence-based abstention. The model emits a confidence alongside each answer; below a threshold, the system declines and hands off to a human. To set the threshold they need a calibration number — how well stated confidence tracks actual accuracy — and they measure it on the base model, the pretrained checkpoint, before any alignment work. The number is good. Pretrained models are reasonably well-calibrated on factual question-answering: when they say 70% they are right about 70% of the time. The team sets the threshold against that curve and moves on.

Then they take the model the rest of the way — supervised fine-tuning, then reinforcement learning from human feedback, the alignment that turns a next-token predictor into a helpful product — and ship it with the abstention logic they validated on the base model.

The abstention logic does not work anymore. Not because the threshold drifted by a few points, but because the quantity it gates on was reshaped by the alignment step. The model that used to say 70% and be right 70% of the time now says 95% and is right 70% of the time — and it says 95% most loudly on exactly the cases it gets wrong. The calibration curve the team validated belongs to a model that no longer exists. RLHF did not just make the model more helpful; it trained the model's verbalized confidence into a systematic liar, for a reason that is structural, documented, and not going away on its own.

This post is about why confidence-as-a-signal evaluations — selective prediction, abstention, confidence-based routing — are invalid specifically on RLHF-aligned models, why the mechanism is the reward model's preference for confident-sounding text, and why a calibration number measured on the base model does not transfer to the model you deploy.

## The base-model calibration number does not survive alignment

Start with the fact that breaks the team's plan, because it is the one most likely to be assumed away. The well-calibrated number they measured was real. It just was not a property of the thing they shipped.

[_Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach_](https://arxiv.org/abs/2505.01997) (arXiv 2505.01997) states it without hedging: _while the pre-trained models are typically well-calibrated, LLMs tend to become poorly calibrated after alignment with human preferences_. The base model's calibration curve and the aligned model's are measurements of two different objects, and the abstention threshold the team set was fit to the wrong one.

The paper is careful about what kind of fact this is. Miscalibration after alignment is not framed as an intrinsic trait of large models — it is _a notable side effect of preference alignment_. The distinction is the whole argument. If overconfidence were intrinsic, you could measure it once, on any checkpoint, and carry the number forward. Because it is a side effect of a specific training step, a number measured before that step cannot account for it. You cannot crash-test the chassis and infer the safety of the finished car.

And the mechanism is not gentle. The same paper traces the miscalibration to _the preference collapse issue in alignment_ that _undesirably generalizes to the calibration scenario, causing LLMs to exhibit overconfidence and poor calibration_. Preference alignment teaches the model to sharply prefer the chosen response over the rejected one; that sharpening — collapse — does not stay confined to the preference task. It leaks into how the model reports confidence, pushing the same all-or-nothing certainty it learned about preferences. The objective that made the model decisive about what to say made it decisive about how sure it is, and decisiveness is not calibration.

## The reward model prefers a confident voice, right or wrong

The previous section says the calibration number dies in alignment. This one names the culprit. It is the reward model.

[_Taming Overconfidence in LLMs: Reward Calibration in RLHF_](https://arxiv.org/abs/2410.09724) (arXiv 2410.09724) isolates the cause, and it is the load-bearing finding of this post: _reward models used for Proximal Policy Optimization (PPO) exhibit inherent biases towards high-confidence scores regardless of the actual quality of responses_. The reward model — the learned proxy for human preference that PPO optimizes the policy against — gives higher reward to answers that sound confident, _independent of whether the answer is correct_. Confidence is rewarded as a presentation style, decoupled from accuracy. Point an optimizer at that reward and you do not get a more accurate model; you get one that learned the cheapest way to earn reward, which is to sound sure. It is the same shape as the failure where [a process reward model grades the fluency of a reasoning step rather than its logical validity](/blog/process-reward-models-fluency/) — a learned proxy, a surface feature it actually rewards, reinforcement learning turning that feature into the objective. Here the surface feature is the sound of certainty, and the casualty is calibration.

The same paper rules out the deflection that this is just the old story about token probabilities. Prior work had shown that _models trained with Reinforcement Learning from Human Feedback (RLHF) are overconfident with a more sharpened output probability_ — a spikier token distribution, established before this paper. What this paper adds is the dangerous part: _RLHF tends to lead models to express verbalized overconfidence in their own responses_. Verbalized. The model does not merely place more probability mass on its top token; it says, in words, that it is highly confident — precisely the signal a verbalized-confidence eval reads, an abstention gate keys on, a confidence router uses to escalate. RLHF corrupts the exact channel these systems depend on.

And the fix confirms where the defect lives. The authors do not offer a decoding trick to read a trustworthy confidence off the deployed model — they propose _two PPO variants: PPO-M: PPO with Calibrated Reward Modeling and PPO-C: PPO with Calibrated Reward Calculation_, and report that both _can reduce calibration error and maintain performance comparable to standard PPO_. The repair has to happen at the reward-modeling and PPO stage, at alignment time, on the objective itself. If the overconfidence could be undone by recalibrating the finished model, you would not need to change how it is trained. That the remedy is a training-time intervention is the strongest evidence that alignment is where the damage is done.

It is fair to ask how a reward model trained on preference comparisons degrades calibration, a quantity nobody put in its training data. The answer is Goodhart: optimizing hard against an imperfect proxy degrades whatever the proxy does not pin down. [_Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?_](https://arxiv.org/abs/2410.05584) (arXiv 2410.05584) supplies the general result — _while there is a weak positive correlation between accuracy and downstream performance, policies optimized towards RMs with similar accuracy can exhibit quite different performance_. A reward model that passes its own held-out accuracy eval gives almost no guarantee about a quantity, like calibration, the eval never looked at. The paper analyzes this _through the lens of the Regressional Goodhart effect_, recognizing that _accuracy, when used for measuring RM quality, can fail to fully capture the potential RM overoptimization_: everything the reward model's notion of a good answer fails to capture is free to drift toward whatever incidentally earns reward. A confident voice earns reward, calibration is not protected, so calibration is spent to buy the voice. The conclusion _underscores the inadequacy of relying solely on accuracy to reflect_ a reward model's _impact on policy optimization_ — and a base-model calibration number is exactly such a metric, fine before deployment, no evidence about what survives optimization.

## Even strong aligned models are not a clean signal

A natural hope is that the biggest, best-aligned models have grown out of this — that frontier RLHF models are calibrated enough to gate on. The evidence says the failure is more textured than "small models bad, big models fine," and the texture is itself the problem.

[_Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Models_](https://arxiv.org/abs/2502.11028) (arXiv 2502.11028) defines the failure precisely — overconfidence is the _misalignment between predicted confidence and true correctness_, and it _poses significant risks in critical decision-making applications_, exactly the deployment the team at the top of this post was building for. Then it complicates the optimistic story. Across nine LLMs and three factual QA datasets, it finds that _large RLHF-tuned models display inherent calibration strengths but can paradoxically suffer increased miscalibration on easier queries_. The strong aligned models are not uniformly miscalibrated — they have real strengths — but they get worse on the easy questions, the ones you would most expect them to be confidently and correctly sure about. An abstention gate assumes confidence is trustworthy precisely where the model is competent; this finding says competence and trustworthy confidence come apart exactly there.

Worse for anyone hoping to validate a single number, the measured value is not stable — it is an artifact of the setup. The same paper reports that structured prompting moved calibration dramatically: _explicitly incorporating distractors can substantially mitigate miscalibration, achieving relative accuracy improvements up to 460% and ECE reductions up to 90%_. A calibration error you can cut by up to 90% by changing the prompt format is not an intrinsic property of the model; it is a property of the model-and-prompt pair, and the number you measure in your harness is not the number you get in production under a different template. The failures also do not average out — the paper identifies _persistent calibration failures, particularly in person-based queries_, non-uniform across question types. A single aggregate number, even an honest one, hides the regions where the model is most dangerously sure of itself.

That this is an RLHF phenomenon and not a generic LLM trait is corroborated independently. [_Confidence Calibration and Rationalization for LLMs via Multi-Agent Deliberation_](https://arxiv.org/abs/2404.09127) (arXiv 2404.09127) opens by tying the problem to its cause: current LLMs are _generally poorly calibrated and over-confident, especially with reinforcement learning from human feedback (RLHF)_. And it treats reading an individual aligned model's own confidence as the thing not to do — existing methods _focus on estimating or eliciting individual confidence_ without leveraging the interaction among multiple models, so the authors reach for an external method instead: _Collaborative Calibration, a post-hoc training-free calibration strategy_ run as a group deliberation among multiple agents, operating _around_ the aligned model precisely because the single model's verbalized confidence is not trustworthy on its own. When the recommended way to get a usable confidence number is to poll several models rather than ask one, the one model's self-report has been judged unreliable.

## What it takes to get a confidence number worth gating on

If the deployed model's verbalized confidence is corrupted, the obvious question is whether you can recalibrate it back to the base-model curve after the fact. The honest answer from the literature is that you mostly cannot — the things that work are training interventions, not decode-time patches.

[_Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models_](https://arxiv.org/abs/2503.02623) (arXiv 2503.02623) starts from the premise that motivates the problem — _a safe and trustworthy use of Large Language Models (LLMs) requires an accurate expression of confidence in their answers_ — the requirement confidence-based abstention and routing try to satisfy, and the one the default aligned model fails. The method does get calibrated confidence out of a model, but look at the cost: it _optimizes a reward based on the logarithmic scoring rule, explicitly penalizing both over- and under-confidence_, to make the model _align its confidence estimates with the actual predictive accuracy_. You have to add an explicit reward that penalizes overconfidence; the default RLHF objective does not contain one, which is why you do not get calibrated confidence by default. And it works by integrating _confidence calibration seamlessly into the generative process of the LLM_ — re-training the model so confidence is part of what it generates, rather than reading a trustworthy number off a model aligned the ordinary way. Calibrated confidence is something you train for, not something you decode.

The downstream application the team built makes the same demand. [_Teaching LLMs to Abstain via Fine-Grained Semantic Confidence Reward_](https://arxiv.org/abs/2510.24020) (arXiv 2510.24020) describes exactly that abstention design: existing methods _fine-tune LLMs to abstain from answering questions beyond their knowledge scope_, but they _rely on coarse-grained signals to guide LLMs to abstain, such as overall confidence or uncertainty scores on multiple sampled answers_. Overall confidence — the aggregate signal RLHF uncalibrates — is the gate, and relying on it gives the model _an imprecise awareness of_ its _own knowledge boundaries_. The fix is again a re-trained signal, not a read of the model as-is: _sample-specific confidence through semantic clustering of multiple candidate answers, training models to retain answers within high-confidence clusters and discard those within low-confidence ones_. The off-the-shelf aligned model's overall confidence is not a reliable gate — the concrete form of this post's claim: the abstention logic the team validated on the base model breaks on the deployed one.

## What this argument does not claim

Five qualifications, because the overstated version loses the reader who knows the field.

First, this is not "calibration is impossible after alignment." Every paper here that diagnoses the failure also presents a repair that works — PPO-M and PPO-C at the reward stage, the log-scoring-rule reward, the semantic-confidence reward, collaborative calibration around the model. The claim is narrower: the _default_ aligned model's verbalized confidence is corrupted, and getting a trustworthy one back takes a deliberate intervention, usually at training time. Calibration is recoverable; it is not free, and it is not the number you measured on the base model.

Second, calibration after alignment is regime-dependent. _Restoring Calibration_ categorizes models into _two regimes: calibratable and non-calibratable, defined by bounds of Expected Calibration Error (ECE)_. Some aligned models can be brought back; some cannot. You cannot assume your deployed model sits in the recoverable regime — you have to measure which one it is, on the deployed checkpoint, the opposite of inheriting the base-model number.

Third, the strongest RLHF models do have genuine calibration strengths — _Mind the Confidence Gap_ says so directly. The failure is not that aligned models are uniformly miscalibrated noise. It is that their calibration is non-uniform across the input distribution, paradoxically worse on easy queries, and movable by up to 90% with a prompt change, so a single aggregate number neither characterizes them nor reveals where they are most overconfident.

Fourth, the headline numbers are anchored to specific models, datasets, and setups. The up-to-460% accuracy and up-to-90% ECE figures are from one paper's distractor experiments — evidence that measured calibration is setup-dependent and movable, not constants to expect on your task. Your model, on your traffic, has its own coefficients, which is the whole argument for measuring on the deployed model rather than importing a number.

Fifth, the mechanism is specifically the reward-model preference plus preference collapse — an RLHF-family artifact, documented for PPO-style RLHF and preference alignment. Whether every alignment recipe induces it to the same degree is not settled here, and this post claims no universal constant across all post-training methods. The durable part is the structure: a learned preference proxy that rewards confident presentation, optimized hard, spends calibration to buy the voice.

## Evaluating confidence on the model you actually ship

The corrective is not exotic: stop treating the base model's calibration as the deployed model's, and evaluate the confidence signal the way you would evaluate any other unmeasured instrument.

**Measure calibration on the deployed checkpoint, never the base model.** Re-measure Expected Calibration Error on the exact aligned model, with the exact prompt template, that goes to production, and determine per _Restoring Calibration_'s two regimes whether your model is even in the recoverable one. A calibration number is valid only for the model-and-prompt pair it was measured on.

**Treat verbalized confidence as a corrupted signal until proven otherwise.** RLHF trains the model to _say_ it is confident regardless of correctness, because the reward model rewards the confident voice. A high stated confidence is, by default, evidence the model learned to sound sure — not that it is right. Validate that the signal tracks accuracy on your task before you let it gate anything.

**Do not gate abstention or routing on the model's overall self-reported confidence as-is.** Coarse aggregate confidence is the exact signal alignment uncalibrates; the literature points at retrained, fine-grained confidence rewards or external multi-model methods instead. This is the same lesson as [a router collapsing onto the expensive model because it was trained to predict a scalar score for what is really a decision](/blog/routers-lose-to-the-baseline/): a signal that looks usable on paper fails because the quantity it reports is not the quantity the decision needs.

**Break calibration down by query type, not one aggregate number.** Failures are non-uniform — worst on person-based queries in one study, paradoxically worse on easy queries in another — so a single aggregate ECE hides the regions where the model is most dangerously overconfident. Report calibration per slice, and set abstention against the worst slices, not the average.

**If you need calibrated confidence, plan to train for it.** The working methods integrate it into the training objective — a reward that penalizes overconfidence, calibrated reward modeling at the PPO stage. Treat trustworthy deployed confidence as a training requirement to design for, not a property you can decode out of an ordinarily-aligned model.

**Keep a human-labeled correctness anchor under the confidence eval.** You can only call a confidence number calibrated against ground-truth correctness on a labeled set. The same discipline that makes [an LLM judge an instrument you measure rather than an oracle](/blog/llm-as-judge-evaluation/) applies here: the model's self-reported confidence is an instrument, and an instrument is only as trustworthy as the labeled set you can hold it against.

## Reading list

- [_Taming Overconfidence in LLMs: Reward Calibration in RLHF_](https://arxiv.org/abs/2410.09724) (arXiv 2410.09724) — pins the root cause: reward models for PPO prefer high-confidence scores regardless of response quality, so RLHF induces verbalized overconfidence, fixable only at the reward-modeling and PPO stage (PPO-M, PPO-C).
- [_Restoring Calibration for Aligned Large Language Models_](https://arxiv.org/abs/2505.01997) (arXiv 2505.01997) — establishes that pre-trained models are well-calibrated and become poorly calibrated after preference alignment, a side effect of preference collapse, with deployed models splitting into calibratable and non-calibratable regimes.
- [_Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in LLMs_](https://arxiv.org/abs/2502.11028) (arXiv 2502.11028) — shows strong RLHF models have calibration strengths yet get paradoxically worse on easy queries, with calibration non-uniform across query types and movable by up to 90% ECE (and up to 460% accuracy) via prompting.
- [_Confidence Calibration and Rationalization for LLMs via Multi-Agent Deliberation_](https://arxiv.org/abs/2404.09127) (arXiv 2404.09127) — independently ties poor calibration to RLHF and treats a single aligned model's self-reported confidence as untrustworthy, motivating a post-hoc, training-free method that works around the model rather than reading its confidence.
- [_Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of LLMs_](https://arxiv.org/abs/2503.02623) (arXiv 2503.02623) — gets calibrated verbalized confidence only by adding an explicit RL reward that penalizes over- and under-confidence, confirming default alignment does not produce it.
- [_Teaching LLMs to Abstain via Fine-Grained Semantic Confidence Reward_](https://arxiv.org/abs/2510.24020) (arXiv 2510.24020) — shows confidence-based abstention built on coarse overall confidence misfires because it gives an imprecise read of the model's knowledge boundary, and needs a retrained fine-grained confidence reward instead.
- [_Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?_](https://arxiv.org/abs/2410.05584) (arXiv 2410.05584) — supplies the general Goodhart mechanism: reward-model accuracy only weakly predicts policy behavior, so optimizing a preference proxy degrades quantities like calibration it never measured, and pre-deployment metrics need not transfer.

The team did the responsible thing — they measured calibration before trusting the model to abstain — and they measured it on the wrong checkpoint. The base model said 70% and meant it; the aligned model says 95% and means whatever earns the most reward, which the reward model decided was the sound of certainty. Confidence-based abstention, selective prediction, confidence routing — all of them read a signal that RLHF specifically corrupts, and a calibration number from before alignment is a measurement of a model you did not ship. Measure the model you ship, on the slices that matter, and treat its confident voice as the thing alignment trained it to produce, not as evidence it is right.