A quality-ascending curriculum beats random shuffling — until a decaying schedule delivers your best data exactly when the learning rate is too small to absorb it. The same coupling flips proxy-model ablations: which dataset wins depends on the schedule, not the data alone.
Two teams own two knobs, and they have never been in the same meeting. The data team owns curation: it scores the corpus and decides what the model sees and in what order. After months of work it sorts the training data in ascending order of quality — lowest-quality shards first, highest-quality last — so the model finishes on its best material. This is curriculum pretraining, and the intuition is old and sound: end on the good stuff. The optimizer team owns the schedule, and it runs the decay everyone runs — a cosine or a warmup-stable-decay that drives the learning rate down to a small fraction of its peak by the end of the run. Neither knob is wrong. Each team tuned its own in isolation, the way the org chart says to.
The data team ships its curriculum and measures it against a random-shuffle baseline. The improvement is small — smaller than the constant-learning-rate ablations promised, small enough that someone asks whether the curriculum was worth the engineering. The conclusion written down is the one the field has written down before: curriculum pretraining gives limited improvements, filed under “tried it, marginal.”
That conclusion is wrong, and the reason is not in the data team’s code. It is in the other team’s schedule. The curriculum delivers its highest-quality shards in the final stretch of training — exactly where the decaying schedule has driven the learning rate down to almost nothing. The best data arrives at the moment the optimizer can take the smallest steps in response. The two are not independent knobs; they are entangled end to end, and the standard schedule is quietly cancelling the curation decision the other team spent months making — after which the cancelled result gets recorded as a fact about the data.
This post is about why a decaying schedule wastes the high-quality tail of a curriculum, why the same coupling flips the verdict of the proxy-model ablations data teams use to choose recipes, and how to stop tuning the two knobs as if they were independent.
The mistake is worth stating precisely. “Curriculum pretraining gives limited improvements” is the conclusion the literature kept reaching. The correction is not “curricula always work” — it is that a curriculum’s measured advantage depends on a knob the experiment held fixed without thinking, at a value that happens to erase the effect.
Start with the part not in dispute: a quality-ascending curriculum has a real, directional advantage. Curriculum Learning for LLM Pretraining: An Analysis of Learning Dynamics (arXiv 2601.21698) pins this down with a control most curriculum papers skip. Pretraining models from 14M to 1B parameters for 300B tokens and running each curriculum reversed, it finds: “direction matters: descending order loses much of the accuracy advantage of the ascending curriculum.” Finishing on the high-quality data is where the accuracy lives — the precondition for everything that follows, because if the order did not matter, the schedule would have nothing to erase.
How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining (arXiv 2511.18903) names the coupling directly: “This work identifies a critical factor constraining these methods: the incompatibility between the ascending data quality order and the decaying learning rate (LR) schedule.” High-quality data at the end of the curriculum, near-zero learning rate at the end of the schedule.
The decisive measurement is what happens when you remove the decay: “while curriculum-based training substantially outperforms random shuffling when using a constant LR, its advantage diminishes under standard LR decay schedules.” Two experiments on identical data; the only thing that changed was a knob the optimizer team owns, and it took the curriculum’s win with it.
The fix is on the schedule side, which is the tell. The paper recovers the benefit not with better data but with two optimizer interventions: “a more moderate LR decay schedule, where the final LR is only moderately smaller than the peak LR,” and “replacing LR decay with model averaging, i.e., computing a weighted average of the final few checkpoints.” Both keep the high-quality tail where the optimizer can still respond. The payoff, on the exact same corpus: “we improve the average score on a suite of standard benchmarks by 1.64% over random shuffling, without additional data refinement,” “validated on 1.5B-parameter models trained over 30B tokens with various data-quality metrics.” That 1.64% was always in the data the team already had; the schedule was sitting on it. The framing to carry away: the findings “underscore the potential of co-designing data curricula with optimization methods.” Co-designing — not tuning in separate meetings.
Why should a high learning rate fail to “absorb” good data and a low one succeed? Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective (arXiv 2410.05192) supplies the picture. Warmup-stable-decay holds the learning rate high and flat, then decays it sharply — and it “generates a non-traditional loss curve: the loss remains elevated during the stable phase but sharply declines during the decay phase.” The gain shows up only when the learning rate comes down, and the explanation is the river valley: a loss landscape like a deep valley with a river along its floor. In the stable phase, “the iterate undergoes large oscillations due to the high learning rate, yet it progresses swiftly along the river” — real distance, but bouncing off the walls, so measured loss stays high. The decay then “minimizes the iterate’s oscillations, moving it closer to the river and revealing true optimization progress” — settling onto the floor turns covered distance into visible loss reduction.
This cuts more carefully than the slogan. A high learning rate does not ignore good data — both phases do real work in different directions, “the sustained high learning rate phase and fast decaying phase are responsible for progress in the river and the mountain directions respectively, and are both critical.” But data in the decayed tail lands when the steps are tiny and along-river motion has mostly stopped, and a quality-ascending curriculum loads its best shards into precisely that tail — not unused, but not bought at full price either.
A quantitative version of the same fact: A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules (arXiv 2503.12811) fits final loss with decay as its own separate term — the law “takes a multi-power form, combining a power law based on the sum of learning rates and additional power laws to account for a loss reduction effect induced by learning rate decay.” A step’s contribution to final loss depends on where in the decay it falls, not only on the data; the law predicts “the loss curves for unseen schedules of different shapes and horizons” from the same data, and optimizing it yields a schedule that “outperforms the widely used cosine learning rate schedule.” The schedule is a knob worth optimizing — the one the data team treats as a fixed constant of the universe.
If the schedule can erase a curation decision in a full run, it can also corrupt the cheap experiments teams use to make those decisions. Data teams do not run a full-scale job to compare two recipes; they train small proxy models and read off which recipe wins, holding the training configuration fixed across every recipe in the name of a fair comparison. That protocol assumes the schedule is neutral. It is not.
Can Small Training Runs Reliably Guide Data Curation? Rethinking Proxy-Model Practice (arXiv 2512.24503) is the second half of the thesis. It names the flaw precisely: “we uncover a subtle yet critical issue in the standard experimental protocol for data recipe assessment: the use of identical small-scale model training configurations across all data recipes in the name of ‘fair’ comparison.” Holding the configuration identical sounds like fairness; it is the source of the error, because the right configuration is not the same for every dataset: “the experiment conclusions about data quality can flip with even minor adjustments to training hyperparameters, as the optimal training configuration is inherently data-dependent.” Flip — not shift, not narrow: the verdict on which dataset is better reverses, because the one-size-fits-all config is wrong for some recipes by enough to invert the ranking.
This is the curriculum result viewed from the evaluation side: there a fixed schedule erased a real data advantage, here a fixed configuration manufactures one that does not survive letting the configuration fit the data. Either way the schedule is part of what is measured — “dataset A beats dataset B” is a property of A, B, and their shared configuration.
The fix confirms which lever is in control, because it is a learning-rate intervention: “a simple patch to the evaluation protocol: using reduced learning rates for proxy model training,” which “yields relative performance that strongly correlates with that of fully tuned large-scale LLM pretraining runs.” Lowering the proxy learning rate restores agreement between the cheap experiment and the run it predicts — direct evidence that the schedule decides whether a data verdict is real or an artifact, “validated across 23 data recipes covering four critical dimensions of data curation.”
The curriculum-dynamics paper adds the scale dependence that makes this dangerous: curriculum effects are largely a small-model phenomenon — “at larger scales, these stability differences are smaller” — and curricula reshape “time spent in each phase” within “a shared sequence of latent phases,” not the phases themselves. Stack the findings: curricula reshape phase timing; where the learning rate sits at each phase decides whether that reshaping is absorbed; and the effect is strongest in exactly the small models teams use as proxies — the regime where a fixed schedule is most able to flip a verdict. The proxy is small, so the coupling is large; its verdict is read as a fact about the data, then shipped to a full-scale run where the interaction is different. The number was never about the data alone.
Two honest qualifications, because the version of this argument that overstates loses the reader who knows the field.
First, this is not “decaying schedules are bad, use a constant learning rate.” The decay phase is not waste in general — both phases are critical, and the multi-power-law paper beats cosine by tuning the decay, not abolishing it. A constant learning rate makes the curriculum’s advantage measurable but is not the production recipe here, and the anchor paper does not offer it either: its fixes are a moderate decay and model averaging, both keeping a decay-like settling step while refusing to drive the late-training learning rate to near-zero. The claim is narrow — an aggressive decay paired with a quality-ascending order lands your best data where the optimizer can least use it. Relieve that specific collision; do not throw out decay.
Second, the magnitudes are scale-bounded, and the bound cuts against the convenient reading. The curriculum-decay result is “validated on 1.5B-parameter models trained over 30B tokens”; the dynamics paper runs 14M to 1B; curriculum stability effects shrink at larger scales — so the 1.64% recovery is a sub-2B-parameter measurement, and frontier-scale behavior is an extrapolation. But that caveat does not rescue the proxy practice it threatens: the whole point of a proxy is that it is small, and “small” is where the coupling is strongest. And 1.64% is a single suite-average from one corpus-and-metric setup — the recovered benefit is real and non-trivial, not a constant you will reproduce on your data.
You cannot read a data recipe’s value off an experiment that fixed the schedule, nor buy a curriculum’s full benefit under a schedule that decays through its best data. Stop treating the two knobs as different problems.
Co-design the curriculum and the schedule; do not tune them in separate meetings. If the data team orders shards by ascending quality, the optimizer team’s decay decides whether that tail lands in a regime the optimizer can use. Set the data order and schedule shape together, against one objective. The same trap — two knobs owned by two teams, jointly optimal but tuned in isolation — is the one that lets test-time compute move the compute-optimal pre-training point without anyone repricing the model size.
When the curriculum ends on its best data, do not decay through it. A quality-ascending order and an aggressive decay are a known-bad pairing. Relieve it with the two measured fixes — a moderate decay whose final learning rate is only moderately below peak, or model averaging over the final checkpoints in place of full decay — so the late shards arrive while the optimizer can still respond.
Never read a data-ablation verdict off a single fixed configuration. A proxy comparison holding one config across all recipes can flip its ranking under a minor hyperparameter change. Re-run the ablation at a second configuration and trust only the verdicts that survive both; adopt the reduced-proxy-learning-rate protocol shown to correlate with fully tuned large-scale runs, and verify that correlation on a few recipes first.
Remember the gain lives in the data you already have. The 1.64% came with no additional data refinement — it was unlocked by the schedule, not by more curation. Before spending another month scoring the corpus, check whether the schedule is sitting on the benefit it already affords. The instinct that decides whether to fine-tune or prompt at all applies: exhaust the cheaper lever before reaching for the expensive one.
Before you conclude a curriculum “gives limited improvements” or trust a proxy verdict:
The data team did everything its mandate asked — ordered the corpus, ended on the best material, measured against a fair baseline — and recorded a marginal result, because the schedule it never controlled had drained the curriculum’s advantage into a decay phase the optimizer team set by default. The verdict was real as a measurement and wrong as a fact: a statement about the data and the schedule together, mislabeled as one about the data. A data-curation decision and a learning-rate schedule are not two knobs on two desks. They are one coupled system, and the schedule decides whether the curation was worth anything. Tune them together, or keep paying full price for data the schedule throws away.
Our taxonomy argued prompt injection is a vulnerability class you contain, not a bug you fix. Here is the quantitative half — against the strongest published defenses, indirect-injection attack success stays high, and for agents that can act it stays alarming.
10 min →
An agent workflow that pauses for human approval and then resumes looks safe — a person clicked approve. But the approval decision and the authority the resumed step runs under are two different objects, and most systems conflate them by carrying the approval as an in-band signal the resumed call trusts. That is a confused-deputy bug: a forged resume, a replayed request, or an approval granted at one gate authorizes an action it was never meant to. The fix is to stop transporting approval and start deriving authority from the trusted approval record at resume time, scoped to the exact suspended step — capabilities enforced at the tool boundary, fail-closed by default.
14 min →
A mic button on a grounded-answer dialog: Whisper for transcription, the unchanged retrieval path, per-sentence TTS for playback. What broke, what it costs, and why the label says turn-based instead of realtime.
6 min →