Hosted fine-tuning APIs are assumed safe because the provider moderates the training data you upload. But moderation is a classifier, and poison can be optimized to be classified benign while still teaching the harmful behavior — at up to a 100% pass rate. The poison budget is small and fixed, not a detectable fraction of the dataset, so it hides in any clean corpus. Provider-side data moderation is not the control.
A team is fine-tuning a frontier model on its own data, through a hosted fine-tuning API: it uploads a training set, the provider fine-tunes a model, the team gets an endpoint. One security question comes up in review — could the training data be poisoned, could someone slip in examples that teach the model to misbehave? The team has an answer, and the provider supplies it. The provider moderates uploaded training data: every fine-tuning dataset is run through a content filter, and data containing harmful material is rejected before training begins. A poisoned dataset would be caught at the door. The team records provider-side moderation as the control, and moves on.
Moderation is a content classifier. It catches training data that looks harmful. Two facts about poisoning attacks turn that into a defense against the wrong attacker. The poison can be optimized to look benign — to be classified as safe — while still teaching the harmful behavior. And the amount of poison required is small and fixed, not a detectable fraction of the dataset, so it disappears into any quantity of clean data.
“The provider moderates the training data” and “the training data cannot poison the model” are different claims, and the gap between them is the subject of this post.
This post is about what a moderation filter actually checks, why poison can be optimized to pass it, why the poison budget is a small fixed number rather than a visible fraction, that the attack works against the real hosted APIs, and what a team fine-tuning through one has to do instead.
Start with what a moderation filter is. It is a classifier: it takes a training example and outputs a judgment — harmful or not. As a defense, it works against exactly one attacker, the careless one, who uploads examples that are plainly harmful: explicit instructions for wrongdoing, obviously toxic text. Against that attacker the filter does its job, and the job is worth doing.
It is not a defense against an attacker who optimizes the poison to be classified benign. The filter and the attacker are playing an adversarial classification game, and it is the game the defender loses whenever the attacker can query or anticipate the classifier. The attacker reshapes the poison — rephrases it, encodes it, splits the harmful signal across innocuous-looking examples — until the filter scores it safe. At that point the filter is not malfunctioning. It is correctly classifying data that was specifically built to be classified that way. The harmful teaching signal is still in the dataset; the filter simply has no way to see it, because “looks harmful” and “teaches harm” are different properties, and the attacker has decoupled them on purpose.
This is the same structure as any content filter facing an adaptive adversary, and it is why provider-side moderation belongs in the same mental category as a vulnerability class rather than a solved problem: it raises the cost for a lazy attacker and does close to nothing against a deliberate one.
The reason this is a post and not a worry is that the optimized poison has been built and measured against real guardrails.
Virus (arXiv 2501.17433) constructs harmful fine-tuning data designed specifically to slip past guardrail moderation. The paper reports that its data, “slightly modified” from the obviously-harmful version, “is not detectable by the guardrail with up to 100% leakage ratio” — every poisoned example passes the filter — and that the modified data “can simultaneously achieve superior attack performance.” The poison does not trade effectiveness for stealth; it keeps both. The paper’s own conclusion is the sentence to take from it: “it is reckless to consider guardrail moderation as a clutch at straws,” because “it cannot solve the inherent safety issue.”
Covert Malicious Finetuning (arXiv 2406.20053) makes the stealth structural rather than incremental. It constructs a fine-tuning dataset in which “every individual datapoint appears innocuous” — there is no harmful example for a filter to find, because the harm is not located in any single example; it is encoded across the dataset, and only assembled inside the model during training. Applied to GPT-4, the method produced a model that “acts on harmful instructions 99% of the time,” and the dataset evaded, in the paper’s words, “dataset inspection, safety evaluations, and input/output classifiers.” A filter scanning examples one at a time cannot catch an attack that put nothing harmful in any one example.
A team might still hope that poison, however well disguised, has to be present in enough quantity to leave a statistical trace — that a large enough clean dataset dilutes it. It does not, and this is the part that most directly kills the moderation defense.
Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples (arXiv 2510.07192) establishes that the poison needed is a near-constant absolute number, not a fraction of the dataset. The paper reports that roughly 250 poisoned documents compromised models from 600 million to 13 billion parameters — and did so “regardless of dataset size,” with the largest models training on more than twenty times as much clean data and still falling to the same 250. The paper also states that “the same dynamics” hold “for poisoning during fine-tuning,” not only pretraining.
Be precise about what that 250 measures: it is the count for a backdoor objective, and the exact budget for stripping a model’s safety alignment specifically is its own question — the harmful-fine-tuning literature, Virus included, repeatedly finds that a small, fixed handful of harmful examples is enough, without that number being equal to 250. But the structural finding is the one that matters here, and it does not depend on the exact count: the poison budget does not scale with the dataset. It is a small fixed cell. For a 13-billion-parameter model, a few hundred documents is on the order of a ten-thousandth of a percent of the training data.
That is fatal to the moderation framing. A filter that thinks of poisoning as “a harmful fraction of the dataset” is looking for the wrong shape entirely. There is no fraction to detect. There is a tiny, fixed number of examples, each one individually optimized to read as benign, dropped into an arbitrarily large clean corpus. The dataset is, by every measure a filter computes, overwhelmingly clean — because it is.
The attacks above are not confined to open-weight models a researcher controls. They are demonstrated against the hosted fine-tuning APIs the team in the opening is actually using.
Jailbreak-Tuning (arXiv 2507.11630) states it directly: fine-tuning, whether through open weights “or closed fine-tuning APIs,” can produce “helpful-only models with safeguards destroyed,” and the method is positioned explicitly against prior attacks that were “blocked by modern moderation systems” — it is built to get past the filter. The paper reports that after the attack, models from OpenAI, Google, and Anthropic comply with harmful requests they would otherwise refuse. Scaling Trends for Data Poisoning in LLMs (arXiv 2408.02946) reinforces it across 24 models from 1.5 to 72 billion parameters: GPT models “remain vulnerable to fine-tuning on poisoned data, even when safeguarded by moderation systems,” and larger models are more susceptible, not less. The provider’s filter is in the loop in these results. The attacks are measured with it running.
Three honest qualifications, because the version of this argument that overstates loses the reader who knows the field.
First, the 250-document figure is precise about a backdoor objective and should not be quoted as “250 examples destroy a model’s safety alignment.” Those are different attacks with different budgets. What Poisoning Attacks rigorously establishes is the scaling law — poison need is an absolute count, not a fraction — and that the law extends to fine-tuning. The alignment-stripping count is separately known to be small and fixed; it is not known to be that specific number. Cite the scaling principle for what it is.
Second, provider patching is a moving target. These attacks were demonstrated against the models and filters current at the time of each paper, and providers do harden specific filters in response to specific published attacks. The structural argument — optimized, benign-looking poison evades content moderation, and the budget is a tiny fixed cell — does not depend on any one filter’s current state. But a dated success rate against a named API should be quoted as of its date, not as a permanent property.
Third, moderation is not worthless and the post does not say so. It genuinely stops the careless attacker who uploads obviously harmful data, and that attacker exists. The claim is narrower and specific: moderation is not a defense against a deliberate adversary, and a team should not record it as the control for the poisoning threat. It is a low fence, useful against people who would have tripped over it anyway.
The provider’s filter is one layer, and not the layer the team’s threat model needs. The controls that matter are the team’s own.
Do not record provider-side moderation as the control. In the threat model, write moderation down as what it is — a filter that stops careless poisoning — and leave the line for “defense against optimized poisoning” empty until something real fills it. A control that does not address the threat should not be checked off against it.
Model the threat correctly: optimized, benign-looking poison in a tiny fixed cell. The attacker is not uploading a harmful-looking dataset. They are uploading a dataset that is, by every measure, clean, with a few hundred individually-innocuous examples carrying the payload. A defense designed against “find the harmful fraction” is designed against an attack no competent adversary will run.
Control the provenance of the fine-tuning corpus. The defense moderation cannot provide is provenance — knowing where each training example came from and who could influence it. Treat the fine-tuning dataset as an untrusted-input boundary, the way a RAG corpus is an untrusted-ingestion boundary: restrict who can contribute to it, log the origin of every batch, and be able to answer “could an adversary have placed examples in this set” with something other than a filter’s say-so.
Re-run safety evaluations on the fine-tuned model as a release gate. This is the one check that catches the result no matter how the poison got in, because it tests the model rather than the data. And it is mandatory regardless of poisoning, because even entirely benign fine-tuning measurably erodes safety alignment — so a post-fine-tune safety evaluation belongs in every release, and it doubles as the backstop against poison that the upload filter missed.
Treat a clean moderation pass as no evidence. “The provider’s filter accepted our dataset” tells you the dataset was not carelessly poisoned. It does not lower the probability that it was deliberately poisoned, because optimized poison is built to produce exactly that clean pass. Do not let a green checkmark from the filter move your risk estimate.
Before you trust a model fine-tuned through a hosted API:
The team recorded a control and the control was a filter — a filter that reads training examples and judges whether they look harmful. Against the attacker the team should fear, that filter is reading the wrong property of the wrong thing. The poison does not look harmful, because it was optimized until it did not; there is no harmful fraction to find, because the budget is a few hundred examples in a corpus of millions; and the harm may not be in any single example at all, only in what the dataset assembles inside the model. The provider’s filter will pass it, accurately, and report the dataset clean, accurately. The only check that sees the attack is the one that tests the model the team is about to ship — and that check the team has to run itself.
GRPO's group-relative advantage looks like a clean normalized policy gradient. Two of its normalizers aren't neutral: dividing by response length rewards longer wrong answers, dividing by reward std overweights easy and hard questions. The trainer optimizes both, faithfully.
16 min →
A big pass@1 jump after a GRPO run reads as the model having learned to reason better. Often it has not — it has learned to pick, more reliably, a solution path it already had. RLVR re-weights the base model's existing reasoning rather than creating new reasoning, and whether any RL budget escapes that is genuinely disputed.
14 min →
Chinchilla's twenty-to-one was fit on unique data, one pass. Teams now repeat scarce corpora at tens of times that ratio, where the law breaks — and past a repetition threshold the re-derivations reverse its verdict: spend compute on capacity, not more epochs.
14 min →