# Reactive autoscaling for LLM inference is a guarantee you will miss the burst.

A traffic spike hits a model-serving deployment at 14:02:31. The autoscaler does exactly what it was configured to do: it watches GPU utilization, and the moment utilization crosses the threshold it requests another replica. The scheduler finds a GPU, pulls the container, starts loading the model. At 14:02:33 the spike is already receding — the burst lasted two seconds, the way bursts do. At 14:02:58 the new replica finishes loading its weights and reports ready. It has nothing to serve. The queue it was provisioned to drain emptied twenty-five seconds ago, every request in it already timed out or served slow, and the SLO is a number in a postmortem. The autoscaler did its job perfectly and missed the event entirely.

This is not a tuning failure — not a threshold set a few points too high, a cooldown that needs shortening, or a metric that needs smoothing. It is a structural mismatch between two timescales. The thing a reactive autoscaler races — a production burst — is measured in seconds. The thing it has to finish before it can help — loading a large model's weights onto a fresh GPU — is measured in tens of seconds at best and minutes at worst. When the floor under your reaction time is larger than the event you react to, reacting faster is not a fix.

This post is about that inequality, why it is structural, and what the two architectures that beat it have in common — neither of which is "react faster."

## The weight load is a latency floor, not a slow path you can optimize away

Start with the number that makes the rest unavoidable. Loading a large model is not slow the way a cold cache is slow; it is slow the way physics is slow. λScale measures it: "it takes over 18 minutes to load a Llama-70B model (140 GB) over a 1 Gbps network" ([_λScale: Enabling Fast Scaling for Serverless Large Language Model Inference_](https://arxiv.org/abs/2502.09922) (arXiv 2502.09922)). If your weights live on network storage and your autoscaler decides — correctly, on a real signal — that it needs another replica, the replica arrives a third of an hour later, long after any burst you would have wanted to absorb by scaling.

The reflex is to put the weights on fast local storage. That helps; it does not save the reactive model. The same paper measures the SSD path: "loading a Llama-70B model from an SSD to a GPU takes over 30 seconds even with optimized implementations" — and generalizes it: "initializing and loading large models within GPU nodes can take anywhere from tens of seconds to minutes to complete." Thirty-plus seconds is the best case most deployments can realistically engineer, still an order of magnitude longer than the burst.

It is worth being precise about why the floor exists, because "the load is slow" invites the belief that a cleverer loader fixes it. PipeBoost decomposes a cold start and finds where the time goes: "even with local high-performance NVMe storage, the model loading time still accounts for an average of 95.2% of the overall time" ([_PipeBoost: Resilient Pipelined Architecture for Fast Serverless LLM Scaling_](https://arxiv.org/abs/2503.17707) (arXiv 2503.17707)). Compute is not the cold start; container startup is not the cold start; moving the weight bytes is the cold start — 95.2% of it on fast NVMe. And the bottleneck is not even disk: stage the weights in host CPU memory and the picture barely changes — "even when the model resides in CPU memory, the time taken to load the model from the CPU to the GPU still constitutes 94.6% of the total." The ceiling is the PCIe link itself — "each GPU device is connected to the CPU through a dedicated PCIe slot (e.g., PCIe 4.0 x16) by up to 32 GB/s" — so load time is bytes over bandwidth, not a quantity you shrink by reacting sooner. You can saturate the link; you cannot beat it.

## The burst arrives faster than the cold start can finish, by construction

The latency floor would be tolerable if bursts were slow. They are not, and that is the second half of the inequality. λScale characterizes production inference traffic as "request rates surging by more than one order of magnitude within just a few minutes," and draws the conclusion in the same breath: "such high latencies prevent platforms from rapidly scaling out to handle highly dynamic workloads, making it impractical to support online inference services with stringent latency requirements." The traffic moves an order of magnitude in minutes; the replica that would absorb it takes minutes to tens of minutes to arrive.

Lay the SLO over those clocks, because the SLO is what turns the miss into a violation. Chiron pins the order that matters: interactive requests have "tight SLOs in the order of seconds" ([_Hierarchical Autoscaling for Large Language Model Serving with Chiron_](https://arxiv.org/abs/2501.08090) (arXiv 2501.08090)). The budget to satisfy a user is the same order as the burst — and an order of magnitude smaller than the cold start you would incur adding capacity for it. The arithmetic does not close: a reactive scale-up that completes in tens of seconds cannot serve a request due in seconds.

HydraServe puts the contrast in the starkest sentence in this literature: "a cold-start instance produces the first token after more than 40 seconds, whereas subsequent generation takes only ∼30ms per token" ([_HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds_](https://arxiv.org/abs/2502.15524) (arXiv 2502.15524)) — the cold start is roughly a thousand times the per-token cost of a warm replica. And the dominant term inside that 40 seconds is the one the previous section identified: because instances run with constrained network bandwidth, "model fetching becomes the primary bottleneck in cold start latency." That bottleneck is the cruel twist: a reactive autoscaler responding to a real spike adds several replicas at once, and they contend for the same network path — HydraServe has to place workers "across GPUs to avoid network contention among cold-start instances" precisely because the load gets slower when you lean on it hardest. And it is not rare: HydraServe notes "bursty traffic patterns are prevalent in both LLM serving workloads," so the expensive path is the common path.

## You are usually reacting to the wrong signal anyway

There is a second defect stacked on the timing one. The signal most threshold autoscalers fire on is a lagging indicator — it tells you the spike happened, after it happened. TokenScale states it: "existing autoscaling policies, often retrofitted from monolithic systems like those in AIBrix and DistServe, rely on lagging indicators such as GPU utilization or coarse-grained request counts. This results in slow reactions to load spikes, leading to significant Time-to-First-Token (TTFT) and Time-Per-Output-Token (TPOT) SLO violations and costly over-provisioning" ([_TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity_](https://arxiv.org/abs/2512.03416) (arXiv 2512.03416)). Stack the two defects and the loop's whole budget goes to the wrong place: utilization crosses its threshold only after the queue has already built, so the clock on the tens-of-seconds load does not even start until you are already behind. A faster threshold trims the detection delay and leaves the dominant load delay untouched.

This is not one paper's grievance. HeteroScale, from a study at production scale, opens flatly: "serving Large Language Models (LLMs) is a GPU-intensive task where traditional autoscalers fall short, particularly for modern Prefill-Decode (P/D) disaggregated architectures" ([_Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference_](https://arxiv.org/abs/2508.19559) (arXiv 2508.19559)). And the right signal was not obvious enough to assume — it had to be discovered, via "a novel metric-driven policy derived from the first large-scale empirical study of autoscaling signals in production." If the correct signal had to come from the first large-scale study of the question, the utilization-threshold default that ships in most stacks is not merely suboptimal — it is the wrong measurement, late.

## The two architectures that actually win share one move: don't cold-start during the burst

If reacting faster cannot work, what does? The literature converges on two answers, the same answer from two angles. Either you scale _before_ the burst off a leading indicator, so the load finishes before you need the capacity — or you never pay the cold start at all, serving the spike from a GPU that is already warm. Both refuse the race instead of running it faster.

The predictive path replaces the lagging signal with a leading one. TokenScale's is token velocity: "we propose Token Velocity, a novel metric that unifies the prefill, network, and decode stages by quantifying their rate of work. As a leading indicator of system backpressure, it enables proactive scaling." The point is the word _leading_: it fires while the queue is still building rather than after, buying back exactly the head start the weight load demands — start the load when backpressure first turns, and it can finish before the SLO is blown. Chiron makes the same move with "the idea of hierarchical backpressure estimated using queue size, utilization, and SLOs." The wins are not marginal: TokenScale "improves SLO attainment from 50-88% to 80-96% and reduces costs by 4-14% over state-of-the-art systems, including DistServe, BlitzScale, and AIBrix," and Chiron "achieves up to 90% higher SLO attainment and improves GPU efficiency by up to 70% compared to existing solutions." The delta comes from changing _when_ you decide, not how fast you load.

The other path refuses the cold start outright: serve the burst from a GPU already holding warm weights, and never spin up a fresh replica inside the spike. TokenScale's convertible decoders do this within one deployment — "convertible decoders allow decoder GPUs to dynamically execute prefill tasks during traffic spikes, creating a rapid-response buffer that absorbs bursts and eliminates the initialization latency of new prefillers." Eliminates, not shortens, because there is no new instance to initialize. Aegaeon generalizes it across many models on a shared pool — "Aegaeon pioneers token-level scheduling, enabling dynamic model switching decisions after each generated token" — repurposing warm GPUs rather than cold-starting a replica per burst. And that still meant attacking the load, not reacting around it: Aegaeon "reduces model switching overhead by up to 97%, ensuring real-time token-level scheduling with sub-second model switching latency." You get per-token switching only once the switch cost is collapsed.

Where you cannot keep everything warm and must accept some cold starts, the move is _still_ to attack the load, not the trigger — overlap and parallelize the transfer so the load window shrinks. HydraServe "overlaps cold-start stages within workers," reducing "cold start latency by 1.7×--4.7×"; PipeBoost pipelines the load across GPUs to "maximize aggregate PCIe bandwidth," cutting the load-plus-first-token stages for a 13B model from 7.9 seconds with ServerlessLLM to 4.8 seconds. Every one of these systems spends its engineering on the load, the warm pool, or the leading signal — never on a faster threshold, because the threshold was never the bottleneck.

## What this argument does not prove

A few honest qualifications, because the inequality is real but its blast radius has edges.

First, the floor is model-and-storage specific and itself a moving target — the thesis is that it exceeds the burst, not that it is any fixed number. "Over 18 minutes" is a 140 GB model over 1 Gbps and "over 30 seconds" is the same 70B from local SSD; both scale with model size and shrink with faster storage. The NVIDIA Run:ai measurements make the smaller end concrete: even an 8B model's readiness is tens of seconds across every tier ("23.18 seconds on S3, 28.28 seconds on IO2 SSD, and 35.08 seconds on GP3 SSD"), and the same blog cuts the other way — default loaders are far from the floor (on IO2 SSD the same 8B loaded "in just 7.53 seconds" with the Run:ai Model Streamer but took "47 seconds" with the "HF Safetensors Loader") ([NVIDIA Technical Blog](https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer/)). So part of any given cold start is self-inflicted, recoverable by fixing the loader first. But the load is bandwidth-bound: the streamer's win comes from saturating the tier ("~1 GiB/s, the max for GP3 SSD," "around 2 GiB/s" on IO2 SSD), so fixing the loader moves you to the floor — still tens of seconds for a large model — it does not move the floor. Where the model is genuinely small, the storage fast, and the bursts slow, the inequality can flip and reactive scaling is fine.

Second, predictive scaling is not free, and this is not a claim that it strictly dominates. A leading indicator that fires too eagerly over-provisions — TokenScale's own framing names "costly over-provisioning" as a failure mode it must avoid — and a warm pool large enough to absorb any burst is, in the limit, over-provisioning with extra steps. The claim is narrower: against sub-second-to-second bursts under a seconds-scale SLO, a purely reactive threshold-and-react policy cannot meet the SLO by construction, so the viable designs are the predictive and warm-pool ones — and _those_ are where the cost/SLO trade-off is worth tuning. Reactive is not one knob worse; it is on the wrong side of the inequality.

Third, the production-scale evidence is an existence proof, not a recipe. HeteroScale, "deployed ... on tens of thousands of GPUs," reports "increasing average GPU utilization by a significant 26.6 percentage points and saving hundreds of thousands of GPU-hours daily, all while upholding stringent service level objectives," and Aegaeon's "core technologies have been deployed in Alibaba Cloud Model Studio ... reducing GPU consumption by 82%." Those prove the problem is solvable in the predictive/warm-pool direction at real scale — not that any specific system fits your stack, model sizes, or traffic shape, so measure your own floor and your own burst before assuming which regime you are in. That a deployed system's cost and SLO drift quietly, on a longer cadence than attention naturally pays, is the subject of [_three months after handoff_](/blog/three-months-after-handoff/); the per-event spend a warm-pool-versus-cold-start decision moves is what [_agent fleet cost observability_](/blog/agent-fleet-cost-observability/) argues you must attribute after the fact rather than estimate up front.

## Reading list

- [_λScale: Enabling Fast Scaling for Serverless Large Language Model Inference_](https://arxiv.org/abs/2502.09922) (arXiv 2502.09922) — establishes the weight-load floor in hard numbers (18+ minutes for a 70B over 1 Gbps, 30+ seconds from SSD) against bursts that surge an order of magnitude in minutes.
- [_PipeBoost: Resilient Pipelined Architecture for Fast Serverless LLM Scaling_](https://arxiv.org/abs/2503.17707) (arXiv 2503.17707) — shows weight loading is 95.2% of cold start on NVMe (94.6% even from CPU memory), bounded by the PCIe link, and cuts it by pipelining the load rather than triggering faster.
- [_HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds_](https://arxiv.org/abs/2502.15524) (arXiv 2502.15524) — quantifies the cold start at more than 40s versus ~30ms/token warm, with weight fetching and co-located contention as the dominant cost, reduced by overlapping stages.
- [_TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity_](https://arxiv.org/abs/2512.03416) (arXiv 2512.03416) — names the lagging-indicator failure and supplies both fixes: a leading-indicator metric, and convertible decoders that absorb bursts from warm GPUs with no new-instance latency.
- [_Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference_](https://arxiv.org/abs/2508.19559) (arXiv 2508.19559) — finds traditional autoscalers fall short for disaggregated serving, derives the correct signal from the first large-scale production study, and validates on tens of thousands of GPUs.
- [_Hierarchical Autoscaling for Large Language Model Serving with Chiron_](https://arxiv.org/abs/2501.08090) (arXiv 2501.08090) — pins interactive SLOs at the order of seconds, shows SLO-blind reactive autoscalers misfire, and recovers large gains with a backpressure (leading) signal instead of raw utilization.
- [Reducing Cold Start Latency for LLM Inference with NVIDIA Run:ai Model Streamer](https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer/) — independent vendor benchmark: readiness is tens of seconds even for an 8B model across tiers, the load is bandwidth-bound, and a streamer is ~6× the naive default.
- [Aegaeon: Effective GPU Pooling for Concurrent LLM Serving on the Market](https://www.alibabacloud.com/blog/alibaba-cloud-boosts-gpu-utilization-with-ai-infrastructure-breakthrough-at-sosp-2025_602623) — a production system that sidesteps the floor by switching warm GPUs between models per token, after cutting switch overhead up to 97% and reducing GPU consumption 82%.

The inequality is the whole post: loading a large model's weights onto a fresh GPU takes tens of seconds to minutes, a production burst lasts seconds, and the second number is smaller than the first. A reactive threshold-and-react autoscaler is, by construction, a bet that you finish the slow operation before the fast event ends — and you cannot, no matter how sharply you tune the threshold. The architectures that hold the SLO do not run that race faster; they refuse it, scaling before the burst on a leading indicator or serving it from a GPU already warm. Reacting faster optimizes the one number that was never the bottleneck.