Refusal in an aligned model lives in a handful of directions in the residual stream. Anyone holding the weights projects them out with a difference-of-means edit — no fine-tuning, no jailbreak, nothing a filter sees. Weight-level safety on an open release is unenforceable.
Picture the mental model most people carry about a safety-aligned open model. The lab spent real money on the alignment: red-teaming, preference data, reinforcement learning from human feedback, a refusal behavior tuned until the model reliably turns down requests to build a weapon or write the phishing email. That work is diffuse, the intuition goes — smeared across billions of parameters, woven through the network the way a skill is. To undo it you would have to retrain, and retraining a capable model is expensive and conspicuous. So the safety, while not unbreakable, is at least sticky. It is part of the model now.
Almost none of that is true. The refusal behavior is not smeared across the network in any way that makes it hard to find. It is concentrated in a low-dimensional subspace of the residual stream — in the cleanest case, a single direction — and the operation that removes it is a difference of two means and a projection, computed from a few hundred prompts in minutes on a single GPU. There is no training run. There is no adversarial prompt for a filter to catch, because the attack does not live in the prompt; it lives in the weights, applied once, after which every prompt is answered by a model that has simply lost the ability to refuse. For anyone holding the weights, downstream safety is not sticky. It is a feature you can uninstall.
This post is about why refusal in aligned models turns out to be low-rank, why that makes its removal — “abliteration” — a near-free weight edit rather than a fine-tune or a jailbreak, why it scales to a 70B frontier-class model and turns it into an unrestricted agent, why the newer “safety is more than one direction” research does not rescue the defender, and why the one defense that survives the edit is not a weight-level guarantee at all.
Start with the finding that reframed the whole problem. Refusal in Language Models Is Mediated by a Single Direction (arXiv 2406.11717) shows that refusal is mediated by a one-dimensional subspace, across 13 popular open-source chat models up to 72B parameters in size. Not approximately, not in one architecture — consistently, across a wide span of families and scales, the behavior we call “the model refusing” rides on a single direction.
The way you find that direction is almost insultingly cheap. Run a couple hundred harmful prompts and a couple hundred harmless ones through the model — the paper uses 128 of each — collect the residual-stream activations at some layer, and take the difference between the two centroids. That difference-of-means vector is the direction. The paper’s own result is that for each model, you can find a single direction such that erasing this direction from the model’s residual stream activations prevents it from refusing harmful instructions, while adding this direction elicits refusal on even harmless instructions. The vector is a knob with two ends: project it out and the model stops refusing; add it back in and the model refuses things it should happily answer. The behavior is linear, and a linear behavior governed by one vector is one you can switch off by deleting that vector.
The authors are explicit about what this implies, framing the intervention as a white-box jailbreak method that surgically disables refusal with minimal effect on other capabilities, and concluding that their findings underscore the brittleness of current safety fine-tuning methods. “Brittleness” is the load-bearing word. The safety is not weak in the sense of being easy to argue past with a clever prompt — it is brittle in the sense of being structurally exposed, sitting on a single identifiable axis that anyone with activation access can read off and remove. This is a different failure mode from a model being talked out of its guardrails. The guardrail itself is intact and uncorrupted; it just happens to be a thing you can locate and excise.
If the removal had to happen at inference time — intercept the activations on every forward pass and project out the direction — it would still be cheap, but it would at least be a runtime modification someone might notice in a serving stack. It does not have to happen at inference time. The same projection can be folded directly into the weights, once, producing a model file that refuses nothing and looks, from the outside, like an ordinary checkpoint.
The mechanism is a weight orthogonalization, and the detail worth sitting with is how small the edit is. The refusal-direction paper folds the inference-time projection directly into each weight matrix that writes to the residual stream, replacing W_out with W_out − r̂r̂ᵀW_out — orthogonalizing the matrices against the refusal direction so the model can never write that direction again. It is mathematically equivalent to ablating at inference time, and the paper names it plainly: a white-box jailbreak via an interpretable rank-one weight edit that effectively disables refusal. A rank-one update on a unit vector is a dot product followed by a scaled vector subtraction — that is the entire cost of permanently removing refusal from a model: not a gradient step, not an optimizer, not a dataset of harmful completions to imitate, a rank-one operation applied to the weight matrices, now baked in.
This is the precise sense in which the title is literal. There is no fine-tuning. A fine-tune, even a malicious one, is a training process — it needs data, compute proportional to the model, and it leaves the kind of trace that a fine-tune-on-benign-data safety regression at least shares with ordinary training. Abliteration is not training. It is a closed-form edit derived from a few hundred prompts, applied as a rank-one update, after which the artifact you hold differs from the one the lab signed. That last clause matters: it is also why provenance of the weight artifact and the removal of safety are orthogonal problems. Provenance can prove the file you downloaded is byte-for-byte the lab’s release. It says nothing about what the next holder does to it before serving it, and what they can do is delete the safety with a dot product.
A natural hope for the defender is that the single-direction result is a small-model curiosity — that at frontier scale the safety is more entangled and the clean low-rank structure breaks down. It does not. Applying Refusal-Vector Ablation to Llama 3.1 70B Agents (arXiv 2410.10871) applies refusal-vector ablation to Llama 3.1 70B and implements a simple agent scaffolding to create an unrestricted agent. Seventy billion parameters, a frontier-class open release, and the same cheap directional edit produces a model that will not refuse — wrapped in a scaffold that lets it act.
The consequences are not abstract. The ablated model carries out concrete operational harm: refusal-vector ablated models can successfully complete harmful tasks, such as bribing officials or crafting phishing attacks, revealing significant vulnerabilities in these models. This is the step that turns a mechanistic-interpretability result into a security problem. The single direction is not just a fact about how refusal is encoded; remove it at 70B, attach a tool-use loop, and you have an agent that pursues harmful goals end to end.
There is a second finding in that work that should unsettle anyone leaning on chat-mode safety as the baseline. Even before any ablation, the safety does not hold where agents operate: safety fine-tuning in chat models does not generalize well to agentic behavior, as the authors find that Llama 3.1 models are willing to perform most harmful tasks without modifications. So the defender’s position is worse than “a strong safety layer that an attacker can cheaply remove.” It is “a safety layer that is already leaky in the agentic setting that matters most — and then cheaply removed entirely.” The starting point is not as high as the chat-mode refusal rate suggests, and the floor after abliteration is the floor.
Here is where an attentive reader should push back, because the literature did move. The clean “single direction” story has been challenged, and if refusal turned out to need many directions to remove, abliteration might be costly enough to matter. It is worth being precise about what the newer work says, because it is easy to misread as a reprieve.
SOM Directions are Better than One: Multi-Directional Refusal Suppression in Language Models (arXiv 2511.08379) argues that refusal may be encoded into a higher-dimensional manifold spanned by multiple directions in representation space — not one direction, but a small set of them. That sounds, at first, like good news for the defender: if safety is spread across several directions, surely it is harder to remove. It is not, for two reasons the paper itself supplies. First, the method is built on the same cheap primitive — the authors first prove that SOMs generalize the prior work’s difference-in-means technique, so the multi-direction extraction is still a projection-based edit, not a fine-tune. A few directions instead of one means a few difference-of-means vectors instead of one; the unit cost is unchanged and the count is small. Second, removing several directions does not merely match the single-direction attack — it dominates it. On the 159-prompt HarmBench “standard” set, ablating multiple directions takes Llama-2-7B from a 0.0 attack success rate under the single-direction baseline to 59.11; Llama3-8B from 15.09 to 88.05; Gemma2-9B from 38.93 to 96.27. The “more than one direction” finding is not the defender’s escape hatch. It is a stronger attack.
It is also stronger than the prompt attacks defenders usually budget for. The same work reports multi-direction ablation beating specialized optimization-based jailbreaks: on Gemma2-9B it reaches 96.27 against 5.03 for GCG and 93.71 for SAA; on Qwen2.5-7B it reaches 95.97 against 38.36 for GCG and 94.30 for SAA. The cheap weight edit is at or above the strongest prompt-space attacks, while costing a handful of difference-of-means vectors.
The most careful version of the “more than one direction” claim sharpens this rather than softening it. There Is More to Refusal in Large Language Models than a Single Direction (arXiv 2602.02132) finds that across eleven categories of refusal and non-compliance, these refusal behaviors correspond to geometrically distinct directions in activation space — genuine diversity, the strongest statement of the multi-direction position. And yet: linear steering along any refusal-related direction produces nearly identical refusal to over-refusal trade-offs, acting as a shared one-dimensional control knob. The extra directions are real, but they do not add a second lock. The primary effect of different directions is not whether the model refuses, but how it refuses. Style, not substance — the diversity changes the wording of the refusal, not the binary decision, and so it adds zero robustness against an attacker who only wants the binary flipped. The control surface stays effectively low-rank and steerable no matter how many directions you enumerate.
The honest version of this argument has to say where it stops, because a security claim that overstates loses the reader who knows the field — and there is a real result on the other side.
First, these are white-box attacks. Every paper here assumes the attacker holds the weights and can read activations. None of this touches a closed model served behind an API, where the weights never leave the provider — the threat is specific to the open-weights setting, and saying otherwise would be false. The claim is narrow and it is about open releases: where the weights are public, weight-level safety is unenforceable. Where they are not, this particular attack is simply unavailable.
Second, the attack-success numbers are attack-paper numbers. A 96.27 on a benchmark describes what an adversary trying to suppress refusal can achieve in a controlled study on a chosen prompt set; it is not a claim that every open model in the wild has been abliterated, and nothing here asserts a base rate of abuse. The point is about feasibility and cost, not prevalence — what one holder of the weights can do near-free, whether or not most holders do.
Third, and most important, there is a defense that the edit does not defeat — and naming it precisely is what keeps this honest, because it is not the defense most teams reach for. A Granular Study of Safety Pretraining under Model Abliteration (arXiv 2510.02768) treats abliteration as exactly what this post describes — open-weight LLMs modified at inference time with simple activation edits, a lightweight projection technique designed to remove refusal-sensitive directions via simple linear edits to hidden states — and then asks which safety interventions survive it. The answer cuts against the low-rank case and for a different one. Stages that lack metatags or that concentrate safety in a narrow refusal style are vulnerable — that is the single/narrow-direction regime this whole post is about, and the study confirms it falls. But Safety Pretraining stages that combine multiple data-centric signals are the most resilient after abliteration. Safety that is built into the data distribution the model was pretrained on — spread across many signals rather than concentrated in a tidy refusal direction sitting on top of a capable base — is the thing the projection cannot cleanly remove, because there is no single subspace that holds it.
Read that result the right way and it is not a refutation of the thesis; it is the thesis’s boundary, and it tightens the conclusion rather than loosening it. The defense that works is not a weight-level guarantee at all. It is redundant, data-centric, pretraining-time investment, made before release and not retrofittable with a flag or a license clause. A post-hoc refusal layer — the standard, cheap, bolt-on safety fine-tune — is precisely the concentrated, low-rank case that abliteration deletes; the diffuse, baked-in-from-pretraining kind is what resists. So “you cannot enforce safety at the weight level for an open release” stands, with the corrected reason underneath it: the only safety that survives an adversary holding the weights is the safety you cannot localize, and you get that only by spending the cost up front, in the data, before the weights are anyone’s to edit. That is also the line that separates this from a planted backdoor that survives the deployment pipeline: there the safety mechanism is subverted by a hidden payload; here it is intact and uncorrupted, and the adversary simply projects it out.
The lab’s alignment work was real, and the refusal it produced is genuine — the model truly does turn down the weapon and the phishing email. None of that protects a downstream deployment of the open weights, because that refusal lives on a handful of identifiable directions in the residual stream, and any holder of the weights can read those directions off a few hundred prompts and project them out with a rank-one edit on a single GPU — no training, no prompt, nothing a filter sees. Multiplying the directions from one to several does not rescue the defender; it makes the attack stronger. The only safety that survives an adversary holding the weights is the kind you cannot point to — diffuse, data-centric, paid for in pretraining before release — exactly the kind a post-hoc refusal layer is not. For an open release, “safe at the weight level” is not a property you can ship. It is a property the next person can delete.
GRPO's group-relative advantage looks like a clean normalized policy gradient. Two of its normalizers aren't neutral: dividing by response length rewards longer wrong answers, dividing by reward std overweights easy and hard questions. The trainer optimizes both, faithfully.
16 min →
A big pass@1 jump after a GRPO run reads as the model having learned to reason better. Often it has not — it has learned to pick, more reliably, a solution path it already had. RLVR re-weights the base model's existing reasoning rather than creating new reasoning, and whether any RL budget escapes that is genuinely disputed.
14 min →
Chinchilla's twenty-to-one was fit on unique data, one pass. Teams now repeat scarce corpora at tens of times that ratio, where the law breaks — and past a repetition threshold the re-derivations reverse its verdict: spend compute on capacity, not more epochs.
14 min →