# You are out of inference capacity because you are out of power, not GPUs.

A capacity planner looks at a row of racks and counts GPUs. The number is clean, it maps to a purchase order, and it is the number every dashboard reports. So when the inference platform runs out of room — when the next deployment will not fit — the reflex is to read that number as the ceiling and to file a request for more cards. The cards arrive on a lead time measured in quarters. Meanwhile the row that was "full" is, in the only unit that physically governs it, half-empty: the breaker feeding it is nowhere near trip, and the chiller pulling heat off it is loafing. The capacity that was unavailable was never about the GPUs. It was about a power and cooling envelope the dashboard does not draw, and an inference workload that leaves far more of that envelope unused than anyone budgeted for.

This is the misconception worth dismantling: that GPU count is the unit of inference capacity. It is not. The unit is the row, and the row is bounded by amps and by the heat a chiller can carry away, not by how many accelerators physically bolt into the rails. An LLM inference fleet, by the structure of how it draws power, sits well below that envelope most of the time — and that slack is real, claimable capacity. You can fit substantially more servers inside the same breaker and the same chiller budget, with tail latency preserved, if you stop counting cards and start scheduling watts. This post is why the headroom exists, how big it is, and the lever that turns it into deployed servers.

## The row, not the GPU, is what fills up

Start with the economics, because they decide which unit is the real one. The total cost of an inference cluster is not the chips. An analysis of datacenter TCO for LLM inference, [_An Inquiry into Datacenter TCO for LLM Inference with FP8_](https://arxiv.org/abs/2502.01070) (arXiv 2502.01070), puts it flatly: "the purchase cost of the networking, cooling, and power infrastructure necessary to maintain the processors is frequently as large as the purchase cost of the chips and may even exceed it." The thing you are buying when you build out inference is, at least half, the power and cooling plant — and the chips are the cheaper rider on top of it.

Once that is true, the unit of capacity follows directly. The same paper draws the consequence: "the per-chip cost of infrastructure is inversely proportional to the number of servers that fit in a single rack, which in turn depends on the power consumption." Read it twice. How many servers fit in a rack is not set by physical slots; it is set by how much power each server draws against the rack's power budget. Lower the draw per server and more servers fit; the per-chip infrastructure cost falls because the expensive plant is amortized across more chips. The rack is a power budget wearing the costume of a physical enclosure, and the number of GPUs it "holds" is an output of the wattage arithmetic, not an input.

This is the seam that a cost dashboard misses even when it is good. Tracking dollars per run, as [cost observability for an agent fleet](/blog/agent-fleet-cost-observability/) argues you must, tells you where the money went after the fact. It does not tell you that the physical thing capping how many runs can exist at once is a breaker. The dollar ledger and the power envelope are different surfaces, and a team can have a flawless ledger while leaving a third of its physical capacity unclaimed because nothing it watches is denominated in watts.

## Inference is where the watts actually go

If power is the constraint, the next question is where the power is spent — training or inference. The operational answer is inference, and it is not close. [_TokenPowerBench: Benchmarking the Power Consumption of LLM Inference_](https://arxiv.org/abs/2512.03024) (arXiv 2512.03024) reports, citing AWS figures on operational LLM lifecycles, that "inference consumes more than 90% of the energy consumption." Training is the dramatic, one-time spend that gets the press. Inference is the standing load that the power plant is actually sized to carry, day after day, for the life of a deployed model. The envelope you are trying to fit servers inside is an inference envelope.

That matters because inference and training draw power in completely different shapes, and the difference is the whole opportunity. Training pins the GPUs near their thermal design power and holds them there; there is no slack to reclaim. Inference does not. And the reason is mechanical, not incidental — it comes from how a transformer actually executes a request, which splits into two phases with opposite power profiles. The characterization study behind the POLCA system, [_Characterizing Power Management Opportunities for LLMs in the Cloud_](https://www.microsoft.com/en-us/research/wp-content/uploads/2024/03/GPU_Power_ASPLOS_24.pdf) (ASPLOS '24), names the split directly: there are "distinct power usage patterns in prompt (spiky) vs. token phase (longer, more stable, and lower)." Prefill, where the model ingests the prompt, is a short compute-heavy burst that spikes power. Decode, where it emits tokens one at a time, is the long stable tail — and it runs at lower power.

That is the crack in the envelope. The phase that dominates wall-clock time in production serving is decode, and decode is the low-power phase. The work that the power plant is sized to carry spends most of its time well under the peak it was provisioned against. TokenPowerBench also makes clear the mix is not fixed — "long prompts raise the prefill share, while large batch sizes increase the decode share," and for Llama3 70B "the jump from 2K to 10K tokens raises energy per token by roughly a factor of three" — so the profile moves with the workload. A static per-GPU power assumption, the kind a capacity plan bakes in, mis-sizes the envelope in both directions. But the central tendency for a production serving mix is clear: decode dominates, decode is cheap, and the envelope has room.

## Decode draws a fifth of the watts the nameplate implies

Here is the number that should change how a capacity plan is written. The phase-aware energy characterization in [_The Illusion of Power Capping in LLM Decode_](https://arxiv.org/abs/2605.11999) (arXiv 2605.11999) measured decode power across four attention architectures on an NVIDIA H200 and found that "decode draws only 137--300 W on a 700 W GPU." A 700-watt accelerator, during the phase that occupies most of production serving, pulls between a fifth and under half of its nameplate. The capacity planner who provisioned the row at 700 watts per card budgeted for a peak the dominant workload phase never reaches.

The reason it never reaches it is structural, and worth stating precisely because it is what makes the headroom durable rather than a lucky artifact of one workload. The same paper: "no cap ever triggers, because memory-bound decode saturates HBM bandwidth rather than compute and leaves power headroom untouched." Decode is bottlenecked on memory bandwidth — moving the model's weights and the KV cache through the chip — not on the compute units that actually burn watts. The arithmetic logic is idling, waiting on memory, for most of decode. Power follows compute, so power stays low. This is the same memory-bound character that [quantization for serving](/blog/quantization-for-serving/) exploits from the accuracy side: a single-stream decode is starved for bandwidth, which is why shrinking the weights buys latency. Here it buys watts. The chip is not working hard enough during decode to draw its rated power, and no amount of traffic changes that as long as decode stays memory-bound.

A second measurement confirms the slack from the latency direction. [_Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling_](https://arxiv.org/abs/2501.08219) (arXiv 2501.08219) finds that the "decode phase dominates inference time (77-91%) and is largely insensitive to GPU frequency." If the phase that owns 77 to 91% of inference time barely cares what clock the GPU runs at, the GPU is being run faster — and therefore hotter and more power-hungry — than the work requires. The same study quantifies it: "reducing GPU frequency from 2842 MHz to 180 MHz achieves an average of 42% energy savings with only a 1-6% latency increase." Roughly forty percent of the energy is being spent buying one to six percent of latency that the dominant phase did not need. That is not a tuning inefficiency. That is a structural over-provisioning of power against a workload that cannot use it.

## The headroom is a deployable-server lever, not a tuning footnote

Slack that nobody acts on is just an idle breaker. The reason this matters is that the headroom converts, directly, into servers you can deploy in a row you already own. The POLCA work, [_POLCA: Power Oversubscription in LLM Cloud Providers_](https://arxiv.org/abs/2308.12908) (arXiv 2308.12908), names the constraint and the move in one breath: "One of the key bottleneck resources in datacenters is power... Power oversubscription improves the power efficiency of these datacenters, allowing more deployable servers per datacenter, and reduces the deployment time, since building new datacenters is slow." Oversubscription means provisioning more servers against a power budget than would fit if every server drew its nameplate peak simultaneously — a safe bet precisely when the workload rarely hits peak.

And inference is exactly that workload. POLCA: "Our deductions align with the data from production LLM clusters, revealing that inference workloads offer substantial headroom for power oversubscription." The headroom is not a hope; it is observed in production clusters, and it is a property of inference specifically — which is why this lever is safe for a serving fleet in a way it would never be for a training cluster pinned at TDP. The concrete result: "we can deploy 30% more servers in the same GPU cluster for inference, with minimal performance loss." Thirty percent more servers, same cluster, same power plant. The ASPLOS characterization that motivates POLCA reports the same figure from simulation — simulating POLCA, the authors "demonstrate that we can deploy 30% more servers in existing clusters with minimal performance loss," and that it "boosts the provisioned server capacity by 30% while incurring minimal power throttling events." The capacity was there the whole time; it was hidden behind a per-GPU peak that the workload does not draw.

The naive way to claim it is to set a power cap and oversubscribe behind it. But the decode measurements above carry a warning the operator has to hear: capping does not bite the phase that matters. The Illusion paper's whole point is that on decode, "no cap ever triggers" — decode already lives below any reasonable TDP cap, so the cap is theater for the dominant phase. The same paper points at the lever that does work: "clock locking Pareto-dominates power capping universally, recovering up to 32% of decode energy at minimal throughput loss." Lock the clock to where decode actually needs it — recall the 2842-to-180 MHz finding — rather than capping a power level decode never approaches. The mechanism that reclaims the energy is frequency control matched to the phase, not a wattage ceiling the workload sails under.

## Scheduling claims the slack without paying the tail-latency tax

Power-capping a whole row uniformly is a blunt instrument, and under stress it is a damaging one. The thermal- and power-aware scheduling work, [_TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms_](https://arxiv.org/abs/2501.02600) (arXiv 2501.02600), shows what the blunt instrument costs in a power emergency: to stay within constraints the naive "Baseline applies uniform frequency caps up to 35% across servers," leading "to significant performance drops," while "TAPAS maintains (or even improves) performance with up to 12% quality impact." A uniform 35% frequency cut hammers every request, including the prefill-heavy ones that actually needed the clock. Scheduling that knows which server is hot and which request is in which phase routes around the constraint instead of taxing everyone equally.

The capacity TAPAS unlocks is larger than the static oversubscription number, because scheduling responds to the live workload rather than betting on an average. It "supports up to 40% more servers without impacts on quality of results while maintaining thermal and power capping below 0.7% of the time." Forty percent more servers in the same envelope, with the row actually hitting its cap less than three-quarters of one percent of the time — which is the empirical statement that the slack was real and the tail latency held. The mechanism is the row-level physics, not the chip's: TAPAS works by "reducing maximum temperature by 17% and peak row power by 23%" through placement and routing. It is not making the chips draw less in aggregate; it is shaving the coincident peaks — the moments when too many servers spike prefill at once — so the row's peak draw and peak temperature drop, and that lowered peak is exactly the slack that lets more servers move in under the same breaker and chiller.

Note the unit in that sentence: peak _row_ power. The thing being managed is the row's envelope, the same envelope the capacity planner mistook for a GPU count at the top of this post. Scheduling treats the row as the unit of capacity and the watt as the currency, which is precisely the frame the GPU-count dashboard cannot express.

## Where this breaks, and where the number is softer than it sounds

The thesis is physical and the measurements are real, but several load-bearing caveats keep it honest.

First, the most striking number — decode at 137 to 300 W on a 700 W H200 — is a measurement on one GPU generation across four attention architectures, not a universal constant. Architectures with heavier per-token compute, different hardware, or aggressive batching that pushes decode toward compute-bound will draw more. TokenPowerBench's own finding that batch size raises the decode share, and that energy per token can triple from 2K to 10K context, is the warning: the slack is real but it is workload-dependent, and a serving mix that is unusually prefill-heavy or runs at very large batch has less of it. The right move is to measure your own decode-phase power against your own traffic, not to import a number from a four-architecture study and bank 30% on faith.

Second, oversubscription is a probabilistic bet, and bets lose. The 30%-and-40% figures come with the qualifier "minimal performance loss" and "below 0.7% of the time" precisely because the failure mode is a correlated peak: a synchronized burst of prefill-heavy requests across many oversubscribed servers can drive the row toward its real limit at the same instant, and then something throttles. The schemes manage this — TAPAS by routing, POLCA by capping — but "manage" is not "eliminate." An operator claiming this headroom is accepting a small, bounded rate of throttling events in exchange for the capacity, and that trade has to be a deliberate, written decision with a monitored throttle-rate budget, not a silent default. The discipline here is the same steady-state-monitoring discipline any production AI system needs: the slack is safe only as long as someone is watching the rate at which the bet loses.

Third, the energy-savings framing and the capacity framing are related but not identical, and it is worth not eliding them. The 42%-energy-at-1-6%-latency result and the up-to-32%-decode-energy-recovery result are about reclaiming energy per request. The 30%/40%-more-servers results are about fitting more servers in a fixed power budget. They share a root cause — decode is over-powered — but reclaiming energy on each request is what _creates_ the budget headroom that lets more servers fit. Treat them as two readings of one fact, not as additive levers you get to stack twice.

Fourth, none of this touches the model itself. The capacity comes from the power profile of inference, not from any change to weights, accuracy, or output quality — which is the deliberate contrast with footprint-reduction levers like quantization that trade accuracy for room. That is a feature: power-aware scheduling and clock locking are reversible, model-agnostic, and quality-preserving in a way a precision change is not. But it also means the lever is bounded by the physics of the envelope. It will not conjure capacity that the breaker and the chiller cannot physically support; it only claims the gap between the provisioned peak and the workload's real draw. When that gap is genuinely closed — a training job lands in the row, or the serving mix shifts compute-bound — the lever is spent, and the next increment of capacity really is a power-plant build, on the slow timeline POLCA names.

## Reading list

- [_POLCA: Power Oversubscription in LLM Cloud Providers_](https://arxiv.org/abs/2308.12908) (arXiv 2308.12908) — establishes power as the named datacenter bottleneck and shows inference's headroom supports deploying 30% more servers in the same cluster with minimal performance loss.
- [_Characterizing Power Management Opportunities for LLMs in the Cloud_](https://www.microsoft.com/en-us/research/wp-content/uploads/2024/03/GPU_Power_ASPLOS_24.pdf) (ASPLOS '24) — the measurement behind POLCA: prompt is spiky and high-power, token generation is longer, stable, and lower, which is the mechanistic root of the headroom.
- [_TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms_](https://arxiv.org/abs/2501.02600) (arXiv 2501.02600) — shows scheduling, not uniform capping, unlocks up to 40% more servers within the same power and cooling envelope while staying capped under 0.7% of the time.
- [_The Illusion of Power Capping in LLM Decode_](https://arxiv.org/abs/2605.11999) (arXiv 2605.11999) — measures decode at 137-300 W on a 700 W H200 and shows TDP capping never triggers on memory-bound decode, so clock locking is the lever that actually reclaims the energy.
- [_Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling_](https://arxiv.org/abs/2501.08219) (arXiv 2501.08219) — finds decode dominates 77-91% of inference time and is frequency-insensitive, so dropping the clock yields ~42% energy savings at 1-6% latency cost.
- [_TokenPowerBench: Benchmarking the Power Consumption of LLM Inference_](https://arxiv.org/abs/2512.03024) (arXiv 2512.03024) — establishes that inference is over 90% of operational energy and that the power profile moves with the workload mix, so a fixed per-GPU assumption mis-sizes the envelope.
- [_An Inquiry into Datacenter TCO for LLM Inference with FP8_](https://arxiv.org/abs/2502.01070) (arXiv 2502.01070) — the economic statement of the thesis: power and cooling cost rivals the chips, and per-chip infrastructure cost is set by how many servers fit per rack, which is governed by power draw.

The next time the inference platform is "out of capacity," check the breaker before you sign the purchase order. The GPU count that looks full is a count of slots, not a count of watts, and a fleet whose dominant phase draws a fifth of its nameplate has left a third or more of the row's real capacity sitting behind an envelope no dashboard renders. Capacity in a datacenter row is amps and heat, not cards — and the operators who measure the watt instead of the GPU are the ones who fit thirty percent more inference into the building they already paid for.