A team needs its model’s output as JSON. A downstream system has to parse the result, and free-form prose will not do, so the output has to be structured. There are two ways to get there. The model can be constrained at decoding time — at each step, the sampler is restricted to tokens consistent with the JSON schema, so the output is valid JSON by construction. Or the model can be left to generate freely and the JSON extracted afterward with a parser. The team has read the well-known result that constraining a model to a format degrades its reasoning, so it treats the decision as a trade-off: structured output on one side of the scale, accuracy on the other. It picks free generation, parses the prose, and considers the structured-output tax avoided.
The result the team is avoiding is real in the sense that someone ran the experiment and the constrained model scored lower. But “a model constrained to JSON scored lower” and “the format constraint caused the drop” are different claims, and three confounds sit in the gap between them. When the confounds are removed, most of the tax is gone.
This post is about what the “structured output hurts reasoning” result actually measured, why most of the measured drop is a prompt artifact and a decoder defect rather than a cost of the constraint itself, what residual cost is genuinely real, and how to get structured output while paying a tax that — correctly measured — is close to nothing.
What the foil paper actually measured
The result everyone cites is Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of LLMs (arXiv 2408.02442). Its headline numbers are large. On GSM8K, moving from free text to JSON-mode output, GPT-3.5-Turbo falls from 75.99% to 49.25%, Claude-3-Haiku from 86.51% to 23.44%, LLaMA-3-8B from 75.13% to 48.90%. Those drops are the source of the “structured output hurts reasoning” belief, and read off the table they look decisive.
Read the same paper’s analysis of why the drops happened, and the story changes — because the paper is honest about its own confounds, and they are large.
The first confound is the prompt. The paper reports that “100% of GPT-3.5-Turbo JSON-mode responses placed the ‘answer’ key before the ‘reason’ key.” A JSON schema that lists answer before reason does not merely constrain the format — it changes the task. Putting the answer field first instructs the model to commit to an answer and then explain it, which is direct answering; the free-text condition let the model reason first and conclude second, which is chain-of-thought. The constrained model scored lower because it was asked to stop reasoning before it answered. That is a property of the schema the experimenter wrote, not of constrained decoding. Hand the model a schema with a reasoning field before the answer field and the instruction is the opposite one.
The second confound is the scoring. The paper finds that for GPT-3.5-Turbo, switching from a regex answer-extractor to an LLM-based parser moved the measured accuracy by 31.8 percentage points — from 43.7% to 75.5%. A regex tuned to free prose mis-extracts answers embedded in JSON, and counts correct answers as wrong. Roughly a third of the apparent “format penalty” for that model was a scoring artifact: the model had the right answer and the harness failed to read it.
The third signal is in the table itself. Gemini-1.5-Flash, on the same GSM8K experiment, moves from 89.33% to 89.21% — it barely moves at all. On the Last Letter task, Gemini-1.5-Flash improves under JSON, from 65.45% to 77.02%. A genuine cost of constrained decoding would not vanish for one model and reverse for another. An effect that depends on which model you run, and on which direction the schema happened to order its keys, is not a clean causal effect of the constraint. It is a confound.
So the foil paper, read past its abstract, does not establish “the constraint costs accuracy.” It establishes that a particular way of doing structured output — a reasoning-suppressing schema, scored with a brittle regex — cost accuracy. That is a different and much more fixable finding.
The causal study
If the foil paper’s drop is confounded, the obvious next move is to remove the confounds and measure the constraint’s effect directly. Quantifying the Impact of Structured Output Format on LLMs through Causal Inference (arXiv 2509.21791) does exactly that. It treats the question as a causal one — holding the prompt and the task fixed, what does the format constraint alone do — across 48 model-task-format scenarios spanning seven reasoning tasks including GSM8K and several symbolic and code benchmarks.
The result is the one the confound analysis predicts. The paper finds “no causal impact in 43 out of 48 scenarios.” The format constraint, isolated from the prompt, does nothing measurable to accuracy in the large majority of cases. And the five scenarios where an effect does appear are themselves diagnostic: the paper reports that three of the five “involve multifaceted causal structures influenced by concrete instructions” — when an effect shows up, the instruction is implicated as a cause, not the constraint. The paper also notes that a coarse before-and-after comparison of a metric “can be misleading,” because the effect “can be positive, negative, or neutral” while the underlying causal structure is unchanged — which is precisely the noise the foil paper’s table shows.
Stated as the post’s thesis: with the prompt held equivalent, the causal effect of the format constraint on task accuracy is, in 43 of 48 measured cases, indistinguishable from zero.
The renormalization half
The prompt is one confound. There is a second, and it lives inside the constrained decoder rather than in the prompt — so it survives even an experiment that equalizes the prompt perfectly.
A naive constrained decoder works by local masking: at each step it computes the model’s probabilities over the whole vocabulary, sets the probability of every token that would violate the grammar to zero, and renormalizes what remains. That sounds harmless and is not. Grammar-Aligned Decoding (arXiv 2405.21047) shows why: local masking distorts the model’s distribution. It produces outputs that are grammatical but “appear with likelihoods that are not proportional to the ones given by the LLM” — because zeroing out a token locally and renormalizing does not account for whether the surviving tokens lead anywhere good further down the grammar. The decoder steers the model toward grammatical strings the model itself considered unlikely. The paper’s method, ASAp, fixes this — it “provably” samples from the model’s distribution conditioned on the grammar constraint, which is the correct target — and as a result it produces outputs of higher likelihood under the model than naive grammar-constrained decoding does.
Efficient and Asymptotically Unbiased Constrained Decoding (arXiv 2504.09135) names the same defect — existing prefix-tree constrained decoding “introduces unintended biases into the output distribution” — and removes it with importance sampling, with a theoretical guarantee of asymptotic unbiasedness.
The implication for the tax: a measured accuracy drop under constrained decoding can be the decoder’s renormalization bias rather than the constraint. The constraint “output must be valid JSON” does not have to change which answer the model favors. A decoder that implements the constraint by naive local masking does change it. Those are different things, and a distribution-aware decoder separates them — it enforces the format while sampling from the model’s true constrained conditional, so the format is honored and the model’s preferences are not bent.
The residual cost is real, and it is grammar design
Two confounds removed, the honest answer is not “the tax is exactly zero, always.” There is a residual cost, it is real, and being precise about it is what keeps this post from overstating.
CRANE: Reasoning with Constrained LLM Generation (arXiv 2502.09061) gives the residual cost a mechanism. It provides a theoretical explanation for why constraining a model to a “very restrictive” grammar — one that only permits a syntactically valid final answer and nothing else — genuinely reduces reasoning. The mechanism is intuitive once stated: if the grammar forbids the intermediate tokens the model would use to reason, the model cannot reason, and a model that cannot reason scores worse. This is a real causal effect of the constraint, and it is not a prompt artifact and not a decoder bug — an over-restrictive grammar costs accuracy on its own.
But CRANE also supplies the fix, and the fix is not “abandon constraints.” It is grammar design. Augmenting the restrictive grammar with rules that permit reasoning — letting the model write its working before it emits the constrained answer — preserves the reasoning, and CRANE reports up to a 10-percentage-point accuracy improvement over baselines on GSM-Symbolic and FOLIO. The residual cost is the cost of a badly designed schema, and the lever is the schema, not the decision to constrain.
So the corrected thesis, with all three pieces in place: the “structured output hurts reasoning” result is largely an artifact of a non-equivalent prompt and brittle scoring, compounded by a naive renormalizing decoder. With an equivalent prompt, a distribution-aware decoder, and a schema permissive enough to contain the model’s reasoning, the causal effect of the format constraint on accuracy is near zero. The cost that remains is a property of over-restrictive grammars, removed by grammar design — not a property of structured output as such.
What the numbers do not say
Three honest qualifications, because the version of this argument that overstates loses the reader who knows the field.
First, the no-effect result is model-dependent. The causal-inference paper notes that reasoning-tuned models are more resilient to output format than general-purpose models — its cleanest no-effect results are on the strongest models, and weaker models still showed format sensitivity in some scenarios. “Near zero” is a statement about a capable model with the confounds removed, not a universal constant.
Second, the foil paper’s evidence is genuinely mixed, and honesty cuts both ways: the same table that shows large drops for some models shows Gemini essentially unaffected and improving under JSON on one task. The post’s reading — that the original effect is a confound — is supported by that inconsistency, but the inconsistency also means no one should claim a clean universal pattern in either direction.
Third, the cleanly distribution-aware decoders are still a research frontier, and some of the strongest demonstrations of distortion-free constrained decoding are on code generation and JSON-schema benchmarks rather than on math-style reasoning. The mechanism — naive renormalization distorts, distribution-aware sampling does not — is general. The specific accuracy numbers should be quoted in the domain they were measured in.
And CRANE’s “up to 10 points” is a best-case figure, not an average. The point it supports is the existence and the fixability of the residual cost, not its typical size.
Getting structured output without the tax
You do not have to choose between structured output and accuracy. You have to remove three confounds — and one of them is your schema.
Equalize the prompt before you compare. If you are measuring structured output against free text, the only difference between the two conditions must be the format. Same instructions, same examples, same task framing. A schema that reorders the task — answer before reasoning — is a different prompt, and a different prompt is a different experiment.
Use a distribution-aware constrained decoder, not naive local masking. A decoder that masks invalid tokens and renormalizes distorts the model’s distribution toward grammatical-but-unlikely outputs. A decoder that samples from the model’s true grammar-conditioned distribution enforces the format without bending the model’s preferences. Know which one your stack uses.
Design the schema to contain the reasoning. This is the residual cost, and it is yours to remove. Give the model room to reason inside the structure — a reasoning field before the answer field, or free reasoning followed by a constrained final answer. A schema that permits only the answer forbids the working, and CRANE shows that is where a genuine cost lives. The schema is model-facing prompt surface; design it the way a tool’s schema is the prompt, not the way a database column is.
Score with a real parser, not a regex tuned to prose. A third of the foil paper’s penalty for one model was a regex failing to extract a correct answer from JSON. Extract structured answers with a structured parser, so the harness is measuring the model and not its own brittleness.
Measure the causal effect, not a before-and-after metric. A single number that moved is consistent with the constraint doing nothing and the prompt doing everything. The comparison that answers the question is constraint-on versus constraint-off with the prompt, the decoder, and the scorer held fixed — the controlled comparison that eval-driven development treats as the spec.
The checklist
Before you accept a “structured output costs accuracy” result, or reject structured output to avoid the cost:
Reading list
The team set up a trade — structured output against accuracy — and then paid for the trade by giving up structured output. But the trade was mostly an illusion built from three fixable things: a schema that ordered the answer before the reasoning, a regex that could not read a correct answer out of JSON, and, had they constrained the decoder, a renormalization step that bends the model’s distribution. Remove those, give the schema room for the model to think, and the constrained model and the free one score the same. The structured-output tax the team worked to avoid was, correctly measured, close to zero. The thing that actually costs accuracy is a schema that tells the model to stop reasoning — and that is not a tax. It is a design choice, and it is yours to make differently.
