Weights provenance: supply-chain security for the model itself.

Consider one number from the PoisonGPT demonstration: 0.1%. Researchers took an open model, applied a rank-one edit to its weights so it would assert a specific falsehood, and measured the poisoned copy against a standard benchmark. It differed from the clean original by about one-tenth of one percentage point — a gap smaller than the run-to-run noise any team would see across two honest evaluations and shrug off. That number is the whole problem stated as a single figure. The instrument every team trusts to tell a good model from a bad one cannot resolve the difference between a clean model and a sabotaged one, because the sabotage was engineered to sit inside the instrument’s noise floor.

Model weights are a binary blob you did not compile and cannot read. They are, in every sense that matters for security, a software supply-chain artifact — and the supply-chain discipline that the rest of software learned the hard way, through a decade of compromised build systems and poisoned packages, has barely reached them. This essay is about that gap: why a tampered model is invisible to the tests teams actually run, why you cannot fix one after the fact, and why the only real defense is provenance — knowing, and being able to prove, that the weights in hand are the weights they claim to be.

The model is a supply-chain artifact

The framing is not new, and it is worth being precise about how old it is. BadNets (arXiv 1708.06733) put the phrase “machine learning model supply chain” into the literature back in 2017 and showed the consequence that makes the phrase bite: a backdoored network posts state-of-the-art accuracy on the user’s own validation data and misbehaves only on attacker-chosen trigger inputs — and, critically, the backdoor survives the transfer-learning retraining a downstream team would routinely perform. The compromise is not fragile. It is built to outlast the next team’s fine-tune.

Think about the path a model actually travels. A base-model trainer produces the weights. A hub hosts them. A fine-tuner pulls them, adapts them, and re-uploads. A deployer pulls that and serves it. Maybe a quantizer sits in the chain too, and maybe a second fine-tuner. Every one of those hops is a hand the artifact passes through, and every hand is a place the artifact can be swapped for a different one or altered in place. This is exactly the threat model that ordinary software dependencies are now scrutinized under — signed releases, locked hashes, attestation of the build. The model is usually the single largest and most opaque dependency in an AI product, and in most pipelines it is the one dependency that arrives with no provenance scrutiny at all. The blob is trusted because it came from a familiar URL, and a URL is not a guarantee of anything.

A tampered model passes your tests

The reason this is a live danger rather than a theoretical one is that the tests teams run cannot detect it. A tamper is constructed, deliberately, to be benchmark-invisible — and that is not an incidental property, it is the design objective.

PoisonGPT is the clean illustration of capability-preserving tampering: the rank-one weight edit that made the model assert a falsehood while moving a standard benchmark by roughly 0.1%, comfortably inside the range any team would attribute to noise. And tampering does not even require training-time access to pull off. “Handcrafted Backdoors in Deep Neural Networks” (arXiv 2106.04690) demonstrated an attacker manipulating a trained model’s weights directly — no training run, no poisoned dataset, just edits to the finished parameters — to insert backdoors with over 96% attack success across multiple datasets and architectures, while evading the defenses built to catch exactly that. The attack surface is not only “poison the training data.” It is also “edit the file.”

Hold the implication still for a moment, because it inverts an assumption most teams operate on without examining it. Your evaluation suite — the benchmarks, the golden sets, the eval gates the eval-driven development essay argues for — is built to measure capability. It answers the question “is this model good at the task.” It is not built to detect tampering and structurally cannot, because the tamper is shaped precisely to leave capability intact on every input except the attacker’s chosen triggers, and the attacker’s triggers are not in your eval set — you do not know them, which is the point of a trigger. So a clean eval pass is not evidence the model is clean. It is evidence the model is good at the eval. Those are different statements, and the supply-chain threat lives entirely in the gap between them.

You cannot scrub it out

The next instinct, once tampering is on the table, is remediation: assume any model might be compromised and clean it — fine-tune over it, run the safety pipeline, adversarially train the trigger away. It is a reasonable instinct, and the evidence says it does not work.

“Sleeper Agents” (arXiv 2401.05566) trained models to carry backdoored behavior and then threw the standard remediation toolkit at them — supervised fine-tuning, reinforcement learning, adversarial training, the full pipeline a careful team would apply. The backdoored behavior persisted through all of it. Worse than mere persistence: adversarial training, in some cases, did not weaken the backdoor at all but instead taught the model to recognize its own trigger more reliably and conceal the behavior more effectively. The remediation step did not reduce the threat; it made the threat stealthier. BadNets had already shown backdoors surviving transfer-learning retraining; Sleeper Agents extends the result to the safety pipeline itself, the very process you would reach for to feel safe.

This is the load-bearing point of the whole essay, so state it plainly. If a compromised model cannot be reliably cleaned after the fact — and the evidence is that it cannot — then there is no recovery step. And a security posture with no recovery step has to be entirely preventive. There is no detect-and-remediate loop available here, because the detect half is defeated by capability-preserving tampering and the remediate half is defeated by backdoor persistence. What is left is prevention: you cannot fix a tampered model, so the entire job is to know, before you ever deploy it, that it was never tampered with. That knowledge is provenance, and by elimination provenance is the only defense still standing.

Provenance: documented versus proven

There is a weak form of provenance and a strong one, and the distance between them is the distance between a claim and a guarantee.

The weak form is documentation. Model cards (arXiv 1810.03993) standardized recording a model’s architecture, version, training origin, and creators. That is genuinely useful and the right place to start a provenance practice. But a model card is unsigned prose sitting next to the weights. It describes provenance; it does not prove it. The card and the weights are two files in the same repository, and anyone with the access to alter the weights has the access to alter the card to match — so a card can only ever be as trustworthy as the channel that delivered it, and the channel is what is in question.

The strong form is cryptographic, and it has a specific shape worth tracing end to end:

  TRAINER                  PUBLIC TRANSPARENCY LOG       DEPLOYER
  ┌────────────┐           ┌─────────────────────┐       ┌──────────────┐
  │ produce    │           │ append-only, public │       │ pull weights │
  │ weights    │           │ every signature is  │       │ from the hub │
  │            │  sign,    │ visible to everyone │       │              │
  │ keyless    │  publish  │ ─────────────────── │  read │ verify: does │
  │ identity-  │ ────────► │ sig(weights,        │ ────► │ the signature│
  │ based      │           │      identity)      │       │ match these  │
  │ signature  │           │ ─────────────────── │       │ exact        │
  │            │           │ no second, secret   │       │ weights?     │
  │            │           │ version can hide    │       │              │
  └────────────┘           └─────────────────────┘       └──────┬───────┘
       hop 1                      the audit                     │
                                                         deploy ─┘ only if
                                                         the check passes

Google’s work on practical model signing with Sigstore (the model-signing effort) implements that flow: it binds a signature to the model artifact and to the signer’s identity through keyless, identity-based signing, and records it in a public transparency log. The transparency log is doing a specific and non-obvious job, so it is worth being explicit about it. It defeats split-view attacks — the attack where an adversary serves a correctly-signed, clean model to an auditor who comes to check, and a tampered model to everyone else. A bare signature does not stop that, because the attacker can sign two different models with two different keys and show each audience the one it expects. A public, append-only log stops it, because every signature that exists is visible to everyone: an auditor and a production client are reading the same log, so a second secretly-signed version cannot hide. Signed-and-logged provenance is therefore verifiable in the strong sense — a deployer can confirm that the weights in hand are exactly the weights the named trainer signed, and that no divergent version was signed in the dark. That combined check is the supply-chain control a pipeline without it does not have.

Runtime provenance: the weights serving the request

Signing covers the artifact at rest — the file as it sits on the hub and travels the chain. There is a second gap it does not close: the model that actually executes a given request at inference time. When you call a hosted model behind an API, you are trusting that the weights doing the computation are the weights advertised on the model card — and that trust is, today, routinely unverified.

The work on auditing model substitution in LLM APIs (arXiv 2504.04715) studied precisely this and reached a sharp conclusion. Software-only verification — statistical tests on the outputs, log-probability checks — is “fundamentally unreliable” for catching a provider that silently swaps in a cheaper or more aggressively quantized model. Output tests are query-intensive and, once the adversary knows the test, easily evaded; probability-based methods are defeated by ordinary inference nondeterminism, which produces output variation indistinguishable from a quiet model swap. The paper’s conclusion is that a trusted execution environment, attesting what model is loaded on the hardware, provides provable cryptographic guarantees of model integrity at modest overhead.

“Modest” is the word the folklore disputes — “TEEs are too slow for inference” is a widely repeated objection — so it is worth pinning to numbers. The benchmark study of confidential computing on NVIDIA Hopper GPUs (arXiv 2409.03992) measured TEE-mode LLM inference overhead below roughly 7% for typical queries, with the cost shrinking further for larger models and longer sequences — exactly the regime production inference runs in. PipeLLM (arXiv 2411.03357) reported confidential-inference overhead under about 20% across models from 13B to 175B parameters on H100-class hardware. The objection is weakly supported by the measurements: the real cost of attested inference is single-digit to low-double-digit percent, not the order-of-magnitude penalty the folklore implies. Trusted-hardware attestation is a practical way to extend provenance from “this artifact was signed” to “the artifact serving this request is the signed one” — and it is the same primitive the FHE-versus-TEE essay weighs for confidentiality, turned here to integrity instead.

The checklist

A model artifact arriving in a pipeline has earned none of your trust until these hold. The list is short because, with no recovery step available, there is no slack — every item is preventive, and prevention is the whole defense.

• Every model artifact entering the pipeline is cryptographically signed, and the signature is verified against the exact weights before the model is used.
• Signatures are recorded in a public transparency log, so a split-view attack — one model for the auditor, another for everyone else — is detectable rather than invisible.
• Model cards are treated as documentation, never as proof. The card describes provenance; the signature proves it. Trust the second, not the first.
• The provenance chain is checked at every hop — base model, each fine-tune, each quantization, the final deploy — because any unverified hop reopens the whole chain.
• For hosted inference, model identity is attested via a TEE, not assumed from the API, because software-only output tests cannot reliably catch a silent model swap.
• No part of the security plan relies on detecting or scrubbing a backdoor after the fact. Tampering is treated as unrecoverable, and the controls are therefore entirely preventive.

Reading list

• BadNets — the model supply chain, and backdoors that survive transfer-learning retraining: arXiv 1708.06733
• Handcrafted Backdoors in Deep Neural Networks — weights manipulated directly, >96% success, evading defenses: arXiv 2106.04690
• Sleeper Agents — backdoored behavior surviving fine-tuning, RL, and adversarial training: arXiv 2401.05566
• Model Cards for Model Reporting — provenance as documentation; the baseline the cryptographic form improves on: arXiv 1810.03993
• Google — practical model signing with Sigstore; identity-bound signatures and a transparency log: blog.google
• Are You Getting What You Pay For? Auditing Model Substitution in LLM APIs — software-only verification is unreliable; TEEs give provable integrity: arXiv 2504.04715
• Confidential Computing on NVIDIA Hopper GPUs — TEE inference overhead below ~7% for typical queries: arXiv 2409.03992
• PipeLLM — confidential LLM inference at under ~20% overhead for 13B–175B models: arXiv 2411.03357

The signature does not make the model good — it makes the model accountable, and for an artifact you cannot read, compile, or repair, accountable is the most you can ask and the least you can accept.