The load-balancing loss in a mixture-of-experts model is a training-time regularizer that stops the router from collapsing — not a promise of even routing at inference. A competently trained MoE ships with deliberately skewed routing, because even routing means the experts never specialized. Expert-parallel serving sized for uniform load under-provisions the hot GPU and mis-budgets tail latency.
A team is deploying a mixture-of-experts model, and the deployment seems to have a shape the architecture hands them for free. An MoE layer holds many expert subnetworks; a router sends each token to a few of them; so the natural way to serve a large MoE is expert parallelism — put the experts on different GPUs and let each GPU process the tokens routed to the experts it holds. The team sizes the cluster the obvious way. There are, say, sixty-four experts and eight GPUs, each GPU holding eight experts, so each GPU should see one-eighth of the tokens. They provision for that: eight GPUs, each scoped to one-eighth of the load.
The provisioning is not naïve. The model was trained with a load-balancing loss — an auxiliary term whose entire stated purpose is to keep routing balanced across experts. Even load is not an optimistic assumption the team is making; it is the documented behavior the training objective was built to produce. The team has read the architecture correctly and sized the cluster by the book.
In production, one GPU sits pinned near full utilization while the others idle, and because every token must clear the MoE layer before the next layer can begin, the whole batch waits on the busiest GPU. Tail latency is set by the hottest expert. The “one-eighth each” sizing was wrong, and it was wrong on the first request.
“Trained with a load-balancing loss” and “routes evenly when served” are different claims. The first is true. The second does not follow from it — and a well-trained MoE specifically arranges for the second to be false. The gap between the two is the subject of this post.
This post is about why a competently trained mixture-of-experts model ships with deliberately skewed routing, why the load-balancing loss neither prevents that nor is meant to, and how to provision expert-parallel serving for the model you actually have.
Start with what the load-balancing loss actually does, because its name oversells it.
The loss is an auxiliary term added to the training objective with a small coefficient, and it penalizes the router for sending too many tokens to too few experts. It exists because without it the router collapses. Early in training a few experts get slightly better than the rest; the router, optimizing the main objective, sends them more tokens; they get more gradient and improve further; and within a short stretch of training a handful of experts carry nearly everything while the rest are dead weight — the model paid for parameters it never uses. The auxiliary loss is the counter-pressure that stops that runaway.
Notice what that description says and does not say. It says the loss prevents collapse. It does not say the loss enforces uniform routing, and the loss is not tuned to. A coefficient large enough to drive every expert to exactly equal load would also drive the experts to be interchangeable — and interchangeable experts are a failed mixture of experts, because the whole reason to build the architecture is that different experts specialize on different things. The coefficient is set where it prevents collapse while leaving the router free to specialize, and at that value the converged router is balanced enough to keep every expert alive and nowhere close to uniform. Bounded, deliberate imbalance is the target the loss is tuned toward — not a defect it failed to remove.
And the loss is a training-time object. There is no auxiliary loss at inference. At serving time the router is a fixed function: it sends each token where its weights say, and nothing rebalances anything, because you cannot adjust parameters on a model you are only running. “The model was trained with a load-balancing loss” is a true statement about the training run. It is not a statement about the forward pass your serving cluster executes — and the forward pass is what your provisioning has to survive.
The reason this is a post and not a hunch is that the imbalance has been measured on a current model, by people building serving systems who needed the real number.
Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts (arXiv 2601.17111) states the point without hedging: “even well-trained MoE models exhibit significantly imbalanced routing,” and that behavior “is arguably natural — and even desirable — as imbalanced routing allows models to concentrate domain-specific knowledge within a subset of experts.” The paper then puts numbers on it. Probing gpt-oss-120b — 128 experts, served with 8-way expert parallelism — on math workloads, it finds one expert position drawing up to 20 percent of the layer’s tokens where an even split would give it about 3 percent, and one GPU carrying 30 to 35 percent of the work where an even split would give it 12.5. That is not a small wobble around uniform. It is a hot expert and a hot GPU, and the team that provisioned for one-eighth each provisioned for a model that does not exist.
The cost of getting it wrong is also measured. Standard expert parallelism, the paper reports, can run 4.6x slower when the skew is extreme — on the order of 95 percent of tokens hitting a single expert — and peak per-GPU memory can grow up to 4x, which makes the skew an out-of-memory risk and not only a latency one. And the imbalance is not a fixed pattern you can characterize once and provision against: the paper notes that “the degree of imbalance changes on a per-batch basis.” The hot expert this batch is not necessarily the hot expert next batch.
One more line from that paper matters for the deployment story: during post-training and inference, “parameter-altering load balancing, like auxiliary losses, is discouraged or not allowed.” The one tool that kept routing in check is a training tool. By the time you are serving, it is gone, and the routing is whatever the converged weights produce.
If imbalance were a training failure, you might hope a better-trained model would route evenly. The training dynamics say the opposite — balance is something a good MoE moves away from as it trains.
Three Phases of Expert Routing: How Load Balance Evolves During Mixture-of-Experts Training (arXiv 2604.04230) tracks load balance across training checkpoints — of OLMoE-1B-7B and OpenMoE-8B — and finds a non-monotone trajectory in three phases: a surge, in which imbalance rises sharply; a stabilization; and a relaxation. The paper’s one-line summary of the trajectory is the part to keep: “early MoE training prioritizes balance while late training prioritizes quality.” A model early in training is being kept balanced because the router is not yet good enough to specialize safely. A model late in training has been let off that leash, because by then specialization is the thing producing quality.
The converged checkpoint you download is on the far side of that trajectory — past the surge, past stabilization, into the regime where the model has traded balance for quality on purpose. And the paper flags something that matters for anyone trying to assess a model before serving it: the trajectory is “invisible to post-hoc analysis of converged models.” You cannot recover it from the final checkpoint. The final checkpoint shows you a skewed router and does not show you that the skew was an earned, late-training state rather than a bug — so a team inspecting only the model it downloaded has no signal that the imbalance is by design. It just looks like imbalance.
Even a team that measures the average skew and re-provisions for it is not done, because the skew moves while the model serves.
PROBE: Co-Balancing Computation and Communication in MoE Inference (arXiv 2602.00509) describes the serving-time dynamics directly: continuous batching and a stream of diverse concurrent requests cause “expert hotspots to migrate abruptly across GPUs.” The hot GPU is not a fixed GPU. It is whichever GPU holds the experts that this moment’s mix of requests happens to favor, and that mix shifts as requests arrive and complete. The paper’s framing is that expert parallelism, by spreading experts across devices, “amplifies execution stragglers” — the layer finishes when its slowest GPU finishes, and a migrating hotspot means the slowest GPU keeps changing identity.
RepetitionCurse (arXiv 2512.23995) shows the same absence from the adversarial side: because there is no load-balancing constraint at inference, a crafted stream of out-of-distribution inputs can drive “all tokens consistently to the same set of top-k experts,” and on Mixtral-8x7B that produces a 3.063x increase in end-to-end latency. This is an attack paper, and it is fair to it to say so — it also notes the router stays “relatively uniform on natural text,” so its extreme numbers are adversarially induced and not a benign-traffic baseline. Cite it for the mechanism it proves, which is the load-bearing one here: there is no balancer at inference, so nothing structural stops load from concentrating, whether the cause is an adversary or just an unlucky batch of ordinary requests.
Three honest qualifications, because the version of this argument that overstates loses the reader who knows the field.
First, there is no clean cross-model “ships with X percent skew” baseline for benign, in-distribution traffic. The sharp figures — 20 percent versus 3, a GPU at 30–35 percent — are from one paper, on gpt-oss-120b, under math workloads, and that paper itself says the imbalance varies per batch. Mixtral, DeepSeek-MoE, OLMoE, and Qwen’s MoEs will each have their own routing fingerprint. The claim is “expect significant, model-specific skew and measure yours,” not “every MoE is skewed by this number.”
Second, “deliberately trades balance for quality” is an interpretation, and a reasonable one, but it is not a clean measured causal result. The Three Phases paper is candid that its congestion-game framework is not a better predictor of routing than a temperature-scaled softmax — it interprets the trajectory rather than out-predicting a simpler model — and its relaxation phase was observed on two older research-scale MoEs, not on current frontier models. Treat the imbalance as consistently observed and well-motivated; do not treat the “on purpose” as proven the way a controlled experiment proves a cause.
Third, not all imbalance is healthy. Both the Salesforce and the training-dynamics papers are careful to separate mild, specialization-driven skew from genuine expert collapse — the failure mode the load-balancing loss exists to prevent, where a model is left with a few overworked experts and a long tail of dead ones. A model can route unevenly because it specialized well, and a model can route unevenly because it trained badly. The provisioning lesson is the same either way, but the diagnosis is not, and a team measuring extreme skew should rule out collapse before calling it design.
You cannot make a well-trained MoE route evenly, and you should not want to — even routing is the signature of experts that never specialized. What you can do is provision for the routing the model actually has.
Measure the routing distribution on your own traffic before you size the cluster. Run the model on a representative sample of your real requests and record per-expert and per-GPU token counts. The uniform assumption — total tokens divided by expert count — is the one number you know is wrong. The measured distribution, with its hot experts, is the input to provisioning.
Size for the hot GPU, not the average GPU. Expert parallelism finishes a layer when its slowest device finishes, so the device that decides throughput and tail latency is the busiest one. Provision memory and compute headroom against the hot GPU’s measured load, and treat the tail, not the mean, as the latency you are buying — the average GPU’s idle time is not a saving you can spend.
Treat the skew as moving, and re-measure under production batching. The hotspot migrates across GPUs as the request mix shifts under continuous batching, so a single offline measurement characterizes one moment. Measure under realistic concurrency, watch the distribution over time, and where it is feasible adopt a serving layer that rebalances dynamically rather than one that assumes a fixed mapping.
Read uneven expert load as expected, and investigate only its extreme. A hot expert is the normal signature of a specialized MoE; it is not an incident. What warrants investigation is collapse-grade skew — a few experts saturated and a long tail near zero — which is a training pathology wearing the same clothes. Decide the threshold that separates “specialized” from “collapsed” for your model, and alert on the second, not the first.
Re-measure on every model swap. Routing skew is a property of the specific trained weights, the way a serving-time quantization profile is — it does not transfer from one MoE to the next, or even from one checkpoint to the next. A cluster sized for the last model’s routing is not sized for this one.
Before you call an expert-parallel MoE deployment correctly provisioned:
The team’s “one-eighth per GPU” was the architecture’s promise read literally, and the load-balancing loss in the training report seemed to back it. But the loss was a regularizer that kept the router from collapsing, applied during training and absent at inference, and a router kept from collapsing is not a router driven to uniform — a router driven to uniform would be a router whose experts never specialized, which is the model nobody set out to train. The skew on that hot GPU is not a defect to be tuned out. It is what a mixture of experts looks like when the mixture worked. Measure it, provision for the hot device, and re-measure when the model changes — because the one distribution you can be sure the model does not produce is the even one you assumed.
Our taxonomy argued prompt injection is a vulnerability class you contain, not a bug you fix. Here is the quantitative half — against the strongest published defenses, indirect-injection attack success stays high, and for agents that can act it stays alarming.
10 min →
A quality-ascending curriculum beats random shuffling — until a decaying schedule delivers your best data exactly when the learning rate is too small to absorb it. The same coupling flips proxy-model ablations: which dataset wins depends on the schedule, not the data alone.
13 min →
An agent workflow that pauses for human approval and then resumes looks safe — a person clicked approve. But the approval decision and the authority the resumed step runs under are two different objects, and most systems conflate them by carrying the approval as an in-band signal the resumed call trusts. That is a confused-deputy bug: a forged resume, a replayed request, or an approval granted at one gate authorizes an action it was never meant to. The fix is to stop transporting approval and start deriving authority from the trusted approval record at resume time, scoped to the exact suspended step — capabilities enforced at the tool boundary, fail-closed by default.
14 min →