Heuristic privacy defenses report safety by beating weak attacks on average-case samples. Audit them honestly and they leak about 10x more — and lose to the provable DP-SGD baseline they never ran, even one whose formal epsilon is vacuous.
A paper crosses your desk with a defense that promises the good deal you have been waiting for: privacy protection with no accuracy tax. It trains on synthetic data, or distills the model into a student, or relaxes the loss, or selects a clever coreset — anything but the differentially private optimizer everyone tells you cripples utility. The evaluation section looks rigorous. It runs a membership-inference attack, reports the attacker’s accuracy near chance, and concludes the method protects privacy. You believe it, because the table says so, and because the alternative — DP-SGD — is the thing you already decided you could not afford.
Here is the misconception that table is built on: that a defense which beats the attack you ran is a defense that beats the attack. The two are not the same claim, and the gap between them is where an entire class of “privacy-preserving” training tricks lives. The attack in the table is usually weak, usually scored on average-case samples where leakage is naturally low. And the table almost never contains the one comparison that would settle the question — the differentially private baseline the authors declined to run because it costs too much, without ever measuring whether their own method bought anything for the price.
This is not a claim that DP is always worth its accuracy cost; the DP fine-tuning essay takes up that cost-benefit call on its own terms. This is a narrower and harder claim about the evidence. Audit these heuristic defenses the way a competent adversary would — worst case, strong attack, head-to-head against a tuned DP-SGD baseline — and they do not hold up. They leak about an order of magnitude more than their own papers reported, and the provable-but-loose baseline they skipped turns out to be the honest floor.
Start with the metric, because everything downstream is a consequence of it. The standard way to report a membership-inference attack is average-case accuracy: across the whole evaluation set, how often does the attacker correctly guess “member” or “non-member”? Near 50% looks like the attacker is guessing. That number gets written down as the privacy score, and a defense that drives it toward 50% gets written up as protective.
The foundational critique of that practice is Membership Inference Attacks From First Principles (arXiv 2112.03570). Its argument is that average-case accuracy is simply the wrong yardstick for a privacy attack. As the paper puts it, these attacks “are currently evaluated using average-case ‘accuracy’ metrics that fail to characterize whether the attack can confidently identify any members of the training set.” Privacy is not an averaging problem: one confidently identified member is a breach, not something diluted by a thousand samples the attacker could not place. The right metric, the paper argues, is the attacker’s true-positive rate at a very low false-positive rate — “their true-positive rate at low … false-positive rates,” on the order of 0.1% — the tail, not the mean.
This reframing is not cosmetic. When the same paper builds an attack designed for that metric — the Likelihood Ratio Attack — it reports the attack “is 10x more powerful at low false-positive rates, and also strictly dominates prior attacks on existing metrics.” Same model, same data; switch the yardstick from average accuracy to low-FPR true positives and the measured leakage jumps by an order of magnitude. Nothing about the model got worse — the earlier number was measuring the wrong thing, and measuring it flatteringly.
Hold onto that, because it is the lever under everything that follows. A defense graded against an average-case attack is graded on the easy region of the distribution, where even an undefended model does not leak much. The worst-case samples — the outliers, the records that sit alone in their corner of the data — are where leakage concentrates and what the average washes out.
Now apply the strong yardstick to the defenses themselves. The most direct treatment is Evaluations of Machine Learning Privacy Defenses are Misleading (arXiv 2404.17399), and the title is the finding. It identifies three recurring flaws in how empirical defenses get evaluated: prior evaluations “fail to characterize the privacy leakage of the most vulnerable samples, use weak attacks, and avoid comparisons with practical differential privacy baselines.” Read those three as a single failure mode — measure the easy samples, hit them with a soft attack, never run the one baseline that would expose the gap.
Their fix is a worst-case audit borrowed from privacy auditing: rather than averaging over a benign dataset, they “inject canary samples that mimic the most vulnerable data, and focus our audit on those canaries only.” Instead of asking how private the model is on a typical record, they manufacture the hardest records — the ones a real adversary would target — and score the attack on those alone. It is the difference between testing a lock against a passerby and against a locksmith.
The result, across five case studies of published empirical defenses — HAMP, RelaxLoss, SELENA, DFKD, and SSL — is blunt: “prior evaluations underestimate privacy leakage by an order of magnitude” — roughly 10x. The tables were artifacts of a weak evaluation, not measurements of real protection. And the comparison the original papers skipped, once run, lands the second blow: “None of the empirical defenses we study are competitive with a properly tuned, high-utility DP-SGD baseline.” None of them beat the baseline they had declined to run.
The pattern repeats in synthetic-data training, where the intuition for privacy is especially seductive. Does Training with Synthetic Data Truly Protect Privacy? (arXiv 2502.12976) examines four paradigms — coreset selection, dataset distillation, data-free knowledge distillation, and diffusion-generated synthetic data — that all share the comforting story that the training data “is visually dissimilar to the private data.” The verdict: “all these methods give a false sense of privacy.” Visual dissimilarity is not privacy; the model still learns the membership signal regardless of whether the pixels look like the original.
What makes this paper land is that it commits to the right metric up front — “Failing to report privacy leakage on the most vulnerable data provides a false sense of privacy … In this work, we always report privacy leakage in the worst case to avoid underestimating the privacy risk” — and then the worst-case numbers arrive. A forgetting-based coreset method reaches 72.94% true-positive rate at 0.1% false-positive rate. Data-free knowledge distillation, 43.38% TPR at the same 0.1% FPR. Those are not near-chance numbers; at a 0.1% false-positive budget the attacker is fingering members the large majority of the time. A method that would have looked private under average accuracy is, on its most vulnerable records, wide open. And the now-familiar conclusion: “DPSGD remains the best defense, achieving a superior trade-off between privacy, utility, and efficiency … None of these empirical methods achieve a better trade-off than DPSGD.”
Two independent groups, two different families of defense, the same shape of result: scored against weak attacks on average-case samples they looked private; under an honest worst-case audit they leak heavily; and the DP-SGD baseline they were positioned to replace beats every one of them.
You might suspect the worst-case story is itself an artifact — that anyone can manufacture a scary number by hand-crafting an attack. The auditing literature answers directly: the audits were not too aggressive; until recently they were far too weak.
Privacy Auditing of Large Language Models (arXiv 2503.06808) makes the point on the canaries — the planted records an audit uses to measure leakage. Standard LLM audits “rely on basic approaches to generate canaries which leads to weak membership inference attacks that in turn give loose lower bounds on the empirical privacy leakage.” Design them properly and the gap is enormous: on Qwen2.5-0.5B, “our designed canaries achieve 49.6% TPR at 1% FPR, vastly surpassing the prior approach’s 4.2% TPR at 1% FPR.” Same model, same false-positive budget; a better-constructed audit is more than 10x stronger. The leakage was always there — the old audit just could not see it.
The same work shows what a good audit does against a model actually trained with DP: the measurement tightens toward the formal guarantee. They “provide a privacy audit of epsilon ~ 1 for a model trained with theoretical epsilon of 4.” That is the tell. A heuristic defense, audited honestly, diverges from its claimed protection by an order of magnitude; a DP model, audited honestly, lands close to its formal bound. One holds up to scrutiny; the other was scrutiny-shaped marketing.
This generalizes past membership inference. In federated learning, Differential Privacy: Gradient Leakage Attacks in Federated Learning Environments (arXiv 2510.23931) runs reconstruction attacks against intercepted gradients and finds the same split. The DP mechanism works: “DP-SGD significantly mitigates the risk of gradient leakage attacks, albeit with a moderate trade-off in model utility.” The high-utility heuristic does not: its PDP-SGD variant “maintains strong classification performance but proves ineffective as a practical defense against reconstruction attacks.” It kept its accuracy and surrendered the data. The lesson is the one this whole essay is built on — “the importance of empirically evaluating privacy mechanisms beyond their theoretical guarantees.” A clean accuracy number is not evidence of protection; until you run the attack, you have measured nothing about privacy.
Here is the objection that deserves the most respect, because it is half right. To keep utility, practitioners run DP-SGD with a large epsilon — and at those values the formal guarantee really is mathematically near-meaningless. So why treat a baseline whose guarantee says almost nothing as the honest floor? If the number on the box is vacuous, the box is just another heuristic wearing a proof.
It is a fair challenge, and the answer is what makes the baseline trustworthy rather than merely conventional. Why Does Differential Privacy with Large Epsilon Defend Against Practical Membership Inference Attacks? (arXiv 2402.09540) takes the puzzle head-on. It concedes the premise without flinching: “the theoretical privacy guarantees of epsilon >= 7 are essentially vacuous.” And yet large-epsilon DP models resist the modern attacks anyway, because the formal epsilon is a worst-case-of-worst-cases quantity. The DP guarantee holds even against an adversary who already knows everything but the one record in question — “it holds even if the attacker already knows the records of all but one person … practical attackers may lack exact knowledge of (nearly all of) the private data.” Real attackers are nowhere near that informed, so the practical protection vastly exceeds what the loose epsilon promises: “a large DP parameter often translates into a much smaller PMP parameter, which guarantees strong privacy against practical MIAs.”
Sit with the asymmetry, because it is the whole argument. DP-SGD’s guarantee is conservative in the safe direction: the formal number assumes a maximally-informed adversary and therefore understates the real-world protection, so the deployed model over-delivers relative to its own paper. The heuristic defenses run the asymmetry the other way — their evaluations assume a weak adversary on easy samples and overstate the protection, so the deployed model under-delivers relative to its own table. One mechanism errs toward claiming too little safety, the other toward claiming too much. That is exactly why a DP baseline with a vacuous epsilon is still the honest floor — not because the proof is tight, but because every way it is wrong is the safe way to be wrong.
This argument has real limits, and pretending otherwise would repeat the sin it accuses others of.
The strongest version of the critique covers a specific set of case studies — HAMP, RelaxLoss, SELENA, DFKD, SSL, and the four synthetic-data paradigms — not literally every paper with “privacy” in the title. “None of these were competitive” is a finding about the defenses that were audited, by groups who chose which to audit. It is strong evidence that the evaluation methodology is broken across the field, not a closed-form proof that no heuristic defense could ever clear a tuned DP-SGD baseline. The honest reading is that the burden has shifted: a new empirical defense should be presumed leaky until it survives a worst-case, strong-attack, DP-baselined audit, because every comparably-examined predecessor has not.
The DP baseline also has to be earned, not assumed. The claim is “a properly tuned, high-utility DP-SGD baseline,” and that adjective is load-bearing in both directions: a carelessly tuned DP-SGD run is a weak opponent, and beating it proves nothing — so a heuristic paper can rig the comparison just as easily by including a bad DP baseline as by omitting one. The discipline cuts the same way for everyone: the baseline must be tuned to its best high-utility operating point before the comparison means anything.
And none of this collapses the cost question. DP-SGD’s “moderate trade-off in model utility” is a real cost, sometimes decisive, and there are deployments where the threat does not justify paying it at all — which is precisely the call the DP fine-tuning essay is about. The argument here is not “always run DP.” It is narrower: if you claim your non-DP method protects privacy, the DP-SGD baseline is the floor you have to clear, and “we skipped it because it is expensive” is a statement about your budget, not your privacy. The most you can honestly say after skipping it is that you do not know whether your defense bought anything.
The corrective is three requirements the methodology papers converge on; a defense that omits any one has not been evaluated, only described.
Report the worst case, not the average. The metric is true-positive rate at a low false-positive rate, on the most vulnerable samples — inject canaries that imitate them if the natural data does not supply them. An average-accuracy number near 50% is consistent with catastrophic tail leakage and tells you nothing about it.
Use a strong attack, and assume the auditor is. Weak canaries and weak attacks manufacture loose lower bounds that flatter the defense; the same audit, strengthened, has repeatedly moved measured leakage by 10x. If the attack is not at least as strong as the strongest published one, the privacy number is an upper bound on the authors’ effort, not a lower bound on the leakage.
Run the DP-SGD baseline, tuned to its best high-utility setting, in the same table. This is the comparison that has, every time it has been run on these defenses, reversed the conclusion. Omitting it is not a neutral choice — it is the choice that lets a leaky method look protective, and after enough repetitions it stops reading as an oversight and starts reading as the method.
The table that says your defense works is measuring the easy samples with a soft attack against no baseline. Measure the hard samples with a strong attack against the DP-SGD run you skipped, and the deal you thought you found — privacy for free — turns out to be the one comparison you never made.
A 99% needle-in-a-haystack number is bought by literal word overlap between the question and the planted fact. Strip the overlap, or make the haystack realistic, and the same model collapses. NIAH does not predict long-context ability.
13 min →
Teams optimize the retriever — chase recall@k, retrieve more documents — and expect answer quality to follow. It does not. Recall and answer accuracy are decoupled, and past a small number of documents, more retrieved context routinely makes the answer worse.
16 min →
RAG corpora ingest documents from sources no one fully trusts, on the assumption that a wrong document is just noise the retriever averages out. It is not noise. A few crafted passages — sometimes one — reliably flip the answer to a targeted question.
14 min →