A membership-inference attack that separates your book from a held-out one is usually reading the calendar, not the weights. A successful attack is not a sound proof — and the null it would need to test against cannot be sampled.
An author is sure their novel was in the training set, and now they have a number to prove it. The pipeline is the one going around: take the book, take a pile of books the model could not have seen, run a membership-inference attack — score every passage by how confidently the model predicts it, threshold the scores — and watch the book land on the “member” side of the line with an AUC well above chance. The attack worked. The book scores like training data; the held-out pile scores like strangers. A complaint gets drafted, an expert report cites the AUC, and the claim is that the model read the book and here is the classifier that detects it.
The classifier is real and the AUC is real. What it detects is the open question. Because of how the experiment was built, the “member” book and the “non-member” pile differ in more than membership: they were written in different years, drawn from different sources, and they do not share the same vocabulary in the same proportions. A classifier handed two piles that differ along several axes learns whichever axis is easiest, and the easiest axis is almost never the subtle imprint of training. It is the calendar. The model assigns lower loss to text that reads like the era and the corpus it trained on, and the attack reports that as membership. Swap in a held-out pile from the same year, the same sources, with matched n-gram statistics, and the AUC collapses toward a coin flip.
This post is about why the wave of “we can tell the model trained on your data” results does not transfer to a copyright audit — why most reported membership-inference success is a temporal- and distribution-shift artifact, why the attacks that genuinely work need access a real-world auditor does not have, and why, even granting an attack that worked, a successful attack is not the thing a legal claim requires.
Start with the part that holds even if every measurement below went the auditor’s way. There is a structural gap between “my attack flagged this document” and “this document was in the training set, to a standard you could defend,” and the gap does not close by making the attack stronger.
Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data (arXiv 2409.19798) makes this the whole argument. A membership attack outputs a score and a decision; what a data owner needs is evidence, and evidence is a hypothesis test. To turn a flag into a claim, “the data creator needs to demonstrate that their attack has a low false positive rate” — that when the document was not in the training set, the attack does not flag it anyway. That is the null hypothesis: the model was trained without this document. And here is the sentence that ends the copyright-audit dream as a black-box exercise: “sampling from this null hypothesis is impossible, as we do not know the exact contents of the training set, nor can we (efficiently) retrain a large foundation model.” Drawing the comparison distribution means retraining a frontier model from scratch with the document removed, repeatedly, and nobody can do that.
This is not a complaint about any one attack’s accuracy. It is a statement about what a sound proof is: a calibrated false-positive rate under a samplable null. Membership inference on a pre-existing document gives you neither. The paper is explicit about what does yield a sound proof: “data extraction attacks and membership inference on special canary data can be used to create sound training data proofs.” Both require the data owner to have done something deliberate beforehand — get the model to regurgitate a long verbatim passage, or plant a unique marker before training. The auditor inspecting an arbitrary published book has done neither, and cannot retroactively.
Keep that as the ceiling. Everything below is how, well before you reach it, the empirical floor gives out too.
The cleanest way to see that a membership attack is reading the calendar is to delete the model from the experiment and check whether the attack still works. If a classifier that never queries the target model separates members from non-members just as well, the separation was never about what the model memorized.
SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It) (arXiv 2406.17975) ran exactly that control. Most sequence-level membership work splits members from non-members by a temporal gap — members old, non-members recent — which means “the membership classifier might learn the temporal gap in language rather than membership, potentially questioning the validity of MIA results obtained.” So they built a blind bag-of-words baseline: a classifier that ignores the target model entirely and looks only at word statistics. On the standard benchmarks it is nearly perfect — AUC 0.994 ± .003 on WikiMIA(128), 0.937 ± .004 on BookMIA, 0.997 ± .002 on the full Gutenberg split — because “a simple bag-of-words classifier can consistently distinguish between members and non-members across all datasets.” A model-free classifier scoring 0.997 is not detecting memorization. It is detecting that the two piles are made of different words, which is what a temporal split guarantees.
Then they fixed the split. On a randomized (IID) split, where members and non-members come from the same pool and differ only in the coin flip that assigned them, the signal evaporates: “the bag of words classifier barely performs better than a random guess,” and — the part that matters for the audit — even the attacks “proposed most recently … do not perform any better than a random guess in this clean setup.” The real attacks collapse alongside the blind baseline. If the genuine signal had been memorization, removing the temporal confound would have left it standing; it left nothing standing. The recommended fix names the access an auditor lacks: “the use of the randomized split of train and test data” and “the injection of randomized unique sequences in LLM training data.” Both are things you arrange before training a model you control — not things available to someone auditing a shipped model over a book published years ago.
The bag-of-words control shows the benchmark is loaded. The next question is what membership inference does once you take the loaded benchmark away, measured directly across model sizes rather than through a proxy classifier.
Do Membership Inference Attacks Work on Large Language Models? (arXiv 2402.07841) is the large-scale version of that measurement — a suite of Pile models from 160M to 12B parameters, across domains. The headline is flat: “we find that MIAs barely outperform random guessing for most settings across varying LLM sizes and domains,” with the project framing it as MIA methods performing near random — under .6 AUC — in most domains. Scaling the model up does not rescue the attack. The paper attributes the difficulty to how LLMs are trained — a very large dataset seen for few iterations, and a genuinely fuzzy member/non-member boundary.
And where the attack does look like it works, the paper locates the reason, and it is the same one: “the apparent success in such settings can be attributed to a distribution shift, such as when members and non-members are drawn from the seemingly identical domain but with different temporal ranges.” Same domain, different years — the copyright-audit setup exactly. Then it pins the mechanism down further: “temporally shifted settings yield MIA performances significantly higher than when members and non-members are from the same temporal range,” and the paper traces that gap to n-gram overlap — the authors “interpret temporal shift as a change in n-gram overlap distribution between the original and temporally shifted non-members,” measuring the natural non-members at a 39.3% average 7-gram overlap against 13.9% for the temporally shifted ones. Recent non-member text shares fewer n-grams with the training distribution than old member text, the loss reflects that, and the attack cashes the difference in as membership. Take the temporal range out and the lift goes with it.
A third paper reaches the same place from the copyright direction. Nob-MIAs: Non-biased Membership Inference Attacks Assessment on Large Language Models with Ex-Post Dataset Construction (arXiv 2408.05968), motivated by the training-data lawsuits directly, names the confounds — MIA “effectiveness is undermined by biases such as time-shifts and n-gram overlaps” — then builds debiased ex-post datasets matched so the easy axes are gone and measures what survives: “MIAs yield results close to random, with only one being effective on both random and our datasets, but its performance decreases when bias is removed.” Even the one attack that worked on both degraded once the bias was stripped, and the debiased sets gave “AUC-ROC scores comparable to those previously obtained on genuinely random datasets.” Three independent groups, three fair comparisons, one result — the inflated AUC was the bias, not membership.
It would be too strong to say membership inference never produces a real signal. It does. The honest claim is narrower and more damning for the audit use case: the attacks that produce a real signal need access a black-box auditor does not have, and the landmark “did the model read your book” result is the one the rebuttal literature flags as confounded.
That landmark is Did the Neurons Read your Book? Document-level Membership Inference for Large Language Models (arXiv 2310.15007). It introduced the task in the form the lawsuits want — “inferring whether the LLM has seen a given document during training or not” — and reported numbers high enough to read as a copyright signal: “an AUC of 0.856 for books and 0.678 for papers” on OpenLLaMA-7B. But read how members and non-members were separated: “by leveraging commonly used data sources for training and the model release date.” The release date — a temporal split, the construction the same group’s later SoK showed a blind bag-of-words baseline can solve without ever querying the model. The canonical positive result and its refutation come partly from the same authors.
The attacks that move the needle on a fair split reach for white-box-style signal. Context-Aware Membership Inference Attacks against Pre-trained Large Language Models (arXiv 2409.13745) is the strong one, and what it exploits is not a context-free fingerprint: “memorization is context-dependent, triggered primarily when the prefix provides insufficient information for accurate next-token prediction.” To model that it needs the model’s internals and a labeled calibration set — the adversary “queries the target LLM with arbitrary token sequences and obtains per-token losses,” and “we sample an α=30% fraction of non-member test data as calibration data.” Per-token losses are not a black-box API output, and a 30% slice of known non-members is exactly the labeled null an auditor cannot assemble. Even with that access the lift is modest: on Pythia-deduped 2.8B it reaches AUC 0.81 on Arxiv against 0.78 for a plain loss attack and 0.77 for zlib, and 0.90 on Github — where it ties zlib’s 0.90 and clears Min-K%‘s 0.87 by three points — on temporally-mixed Pile domains, the setup the rebuttals flag. A few points of AUC on one domain and none on the other, bought with per-token losses and labeled calibration data on a confounded split, is not a legal proof.
The auditing literature that achieves a sound result makes the access requirement unmistakable. Privacy Auditing of Large Language Models (arXiv 2503.06808) builds the strongest canaries to date, and a canary is something you insert: “a canary is the concatenation of two sequences of tokens: a prefix and a secret both sampled from some randomness,” and “we will be injecting these canaries into model training for auditing.” Even purpose-built canaries yield only a modest true-positive rate at a meaningful false-positive rate — “on the Qwen2.5-0.5B model, our designed canaries achieve 49.6% TPR at 1% FPR, vastly surpassing the prior approach’s 4.2% TPR at 1% FPR.” That prior 4.2% is the regime an auditor without injected canaries is in. The paper frames its result as a first: “this is the first time that a privacy audit of LLM training has achieved nontrivial auditing success in the setting where the attacker cannot train shadow models, insert gradient canaries, or access the model at every iteration.” Every prior “success” presupposed exactly that access — which a copyright auditor inspecting a finished model never has. Even here the bound is loose: the method provides “a privacy audit of ε ≈ 1 for a model trained with theoretical ε of 4.”
A post that has spent five sections demolishing membership inference as copyright evidence owes the strongest version of the other side, because the skeptical case has a soft edge, and pretending otherwise would be the same overclaim it is criticizing.
The soft edge: “membership inference cannot prove training” is about black-box inference on arbitrary pre-existing documents — not a claim that the model is innocent or the data owner out of options. The same paper that supplies the ceiling says so by naming what does work. If a model can be made to emit a long, near-unique passage from a copyrighted work verbatim on demand, that extraction result is much closer to evidence than any membership score, because its null (“a model that never saw this reproduces it verbatim by chance”) is one you can argue quantitatively. Memorization is real; the argument is not that models forget their training data, but that membership inference on a document you did not pre-mark is the wrong instrument for proving it.
A second limit on the skeptics’ own position: the rebuttal results are about average-case documents on randomized splits, and a plaintiff is not obligated to argue the average case. A work duplicated thousands of times across the web, or one with a passage the model reproduces near-verbatim, can be an outlier the model genuinely memorized — and for that document, an attack might carry signal that survives debiasing. The IID-collapse results bound what you can claim in general; they do not certify that this model did not memorize this heavily-duplicated book. The burden is the data owner’s: show the signal survives a same-period, same-source, n-gram-matched null for the specific document, not an AUC borrowed from a temporally-split benchmark.
And the adversarial direction makes even a surviving signal fragile where it has to hold — actual litigation, where the accused is not a cooperative benchmark. On the Evidentiary Limits of Membership Inference for Copyright Auditing (arXiv 2601.12937) asks the courtroom question directly — “whether MIAs can serve as admissible evidence in adversarial copyright disputes where an accused model developer may obfuscate training data while preserving semantic content” — and shows the signal does not survive a developer who fights back. Its SAGE framework paraphrases training data to alter lexical structure while keeping meaning, and “state-of-the-art MIAs degrade when models are fine-tuned on SAGE-generated paraphrases, indicating that their signals are not robust to semantics-preserving transformations.” The conclusion is stated for the legal setting: these results “suggest that MIAs are brittle in adversarial settings and insufficient, on their own, as a standalone mechanism for copyright auditing of LLMs.” The skeptical case is not “the model is clean.” It is that membership inference is the wrong proof, the right proofs need pre-arranged control, a surviving signal must clear a per-document null the auditor has to construct, and a motivated developer can paraphrase the lexical signal away while keeping the content.
You cannot prove training-set inclusion with a membership attack on a document you did not mark in advance. You can stop treating the AUC as evidence, build the comparison the claim actually requires, and reach for the instruments that yield a sound proof.
Reproduce the attack against a matched null before you cite it. The number that means something is not the AUC against a held-out pile of recent books. It is the AUC against non-members from the same period, the same sources, with matched n-gram statistics — the debiased, ex-post construction. If the signal survives, you have something; if it collapses toward chance, you measured the calendar. Run the blind bag-of-words baseline too: if a classifier that never queries the model scores as high as your attack, your attack added nothing.
Treat per-token-loss and calibration-set requirements as disqualifying for a black-box claim. The attacks that beat a fair split need internals and a labeled non-member set. If your evidence depends on per-token losses or a 30% slice of known non-members, it depends on access you do not have against a deployed model — and the opposing expert will say so.
Prefer extraction to inference when the goal is a proof. A long, near-unique passage reproduced verbatim on demand is a stronger object than a membership score, because its null is arguable. If you control the model and want a real audit signal going forward, the sound construction is an injected canary — a unique marked sequence placed before training — not a post-hoc attack on arbitrary text. And if a work is heavily duplicated or partly reproduced near-verbatim, argue that, with a per-document null, rather than a benchmark AUC.
Assume the lexical signal can be paraphrased away. A developer who rewrites training data while preserving meaning degrades the attack while keeping the content. An audit strategy resting on lexical membership signal has a defeat that costs the other side a single paraphrase pass.
This sits next to three failures the rest of this series documents. An attacker with only query access can reconstruct a private corpus one answer at a time — your RAG corpus is extractable, extraction as a capability, the family the sound-proof construction draws on, pointed the other way. Bounding what the weights memorize before any of this becomes a dispute is the job of differential privacy in fine-tuning. And the mirror-image error — assuming a model can simply forget a flagged document on request — is unlearning is not deletion: proving data was in and proving it was removed are both harder than the demo suggests, for related reasons.
The classifier that flagged the book is not lying; it is answering a different question than the lawsuit asks. Handed an old book and a recent pile, it found the difference — and the difference was the year, the source, and the vocabulary, not the imprint of training. Strip those away, same period and sources and n-grams, and the attack falls to a coin flip, which is the literature’s repeated finding on fair splits. Even if a signal survived, a proof is a false-positive rate under a null you cannot sample for a model you cannot retrain. The honest instruments are verbatim extraction and a canary you planted before training, and a copyright auditor inspecting a published book has neither. The AUC is a clock. Do not read it as a fingerprint.
Unlearning promises a model can forget specific data without a full retrain. What it delivers is output suppression: a small benign relearning set — or simply quantizing the model — brings the 'forgotten' knowledge back.
14 min →
The EU AI Act requires AI output to be marked in a robust, reliable way. A formal impossibility theorem says strong watermarking cannot exist. Five words in the statute carry the collision.
13 min →
Redacting the transcript and anonymizing the speaker feels like the privacy work is done. It is not: the waveform is a biometric giving up age, accent, emotion, and clinical conditions from the signal — and it leaks hardest for the groups least seen in pretraining.
17 min →