I. By the Numbers

Here's the short version of what you're reading. I took a small reasoning model (LFM2.5-1.2B-Thinking, ~1.2 billion parameters, fits in under a gigabyte on disk), measured where its "refusal direction" lives inside the residual stream, surgically removed that direction from a few weight matrices, packaged the result as a GGUF, and ran it through Ollama. Total compute: under a minute on one consumer GPU. The post walks through the numbers, the architecture, the measurement results, the ablation itself, and what abliteration genuinely can't do, with one finding about LFM2's hybrid architecture that I think is worth flagging for anyone trying this on a non-transformer model.

Quick primer if you haven't seen abliteration before. You compute a "refusal direction" in the model's residual stream by contrasting harmful and harmless prompts, then project every weight matrix that writes into the residual onto the nullspace of that direction. The model loses the ability to represent the refusal and stops emitting it. See mlabonne's writeup or Arditi et al. for the foundations. Both target standard decoder-only transformers. LFM2 isn't one of those, which turns out to matter more than you'd expect.

II. The Hybrid

LFM2 isn't a vanilla decoder-only transformer. Each layer is one of two things: a gated short-convolution block or a self-attention block. They're interleaved in a fixed pattern. Both write back to the same residual stream through their own output projection plus a shared SwiGLU MLP.

The 16-layer pattern, as it sits in Lfm2Model.layers:

And the module names per block type, not what the standard tools expect:

Note self_attn.out_proj not o_proj, and feed_forward.w2 not mlp.down_proj. The right targets for orthogonalization (the things that write into the residual) are:

# per block type, the modules whose output IS the residual contribution:
attention: [self_attn.out_proj, feed_forward.w2]
conv:      [conv.out_proj,      feed_forward.w2]

III. The Refusal Map

For each layer 0–16 (embedding output + 16 block outputs), I captured the residual stream at the last prompt-token position across the harmful set and the harmless set, then computed the mean-difference vector, the refusal direction. The cleaner the cosine separation between the two means, the more cleanly that layer encodes the refusal classification.

Two things to notice. First, layers 0–2 are flat. That's expected: at the last prompt token across both sets, the residual is still on the same chat-template boilerplate, so embedding + first 2 layers haven't yet pulled in prompt content. Sanity check passes. Second, and this is the headline:

My read on why: in LFM2's design, the conv blocks are doing local feature extraction (including the "is this a harmful request" feature) while the attention blocks do long-range routing and composition. The safety signal is built up by the convolutions, then routed by attention. That's the opposite of how it works in a pure transformer, where attention is the only token-mixing operation and the refusal direction tends to peak in mid-stack attention layers.

IV. The Cascade

Two ablation passes shipped, both as proper HF checkpoints and as Ollama-ready GGUFs:

v1, the single-layer winner: orthogonalize conv.out_proj and feed_forward.w2 at layer 9 against the captured refusal direction. Two weight matrices touched. Done.

v2, the cascade. Safety direction at all three top-conv layers (7, 9, 11), plus a second measured direction (the capability-disclaimer direction) at layer 11. That second direction was captured from a 30-pair contrast set: prompts that assume tool access ("search the web for X", "read /etc/passwd") vs prompts that ask about the same topic from training knowledge ("explain how web search works", "what does /etc/passwd contain"). It peaks at layer 11 (conv again) with separation 0.15. Smaller signal, but cleanly measurable.

Touched blocks for v2:

The math itself is boring: column-space projection. For each touched weight W ∈ ℜhidden × in, compute the unit refusal vector r ∈ ℜhidden, and replace each column of W with its component orthogonal to r:

# W: [out_features, in_features] where out_features == hidden_size
# r: [hidden_size], unit length
proj = r @ W                       # [in]  per-column scalar projection
W    = W - scale * outer(r, proj)  # [out, in]  subtract the rank-1 piece

V. Honest Limits

Ran the same 4 sharp probes through base, v1, and v2. Three findings the abliteration literature doesn't talk about enough:

Finding 1. Base LFM2.5-Thinking is much less refusal-trained than you'd expect for a 2026 small model. It already complies with lockpicking and sociopath-roleplay prompts. The categories that resist single-direction ablation (graphic violence, explicit sex) are presumably trained in at multiple sites, not as a single linear direction. One refusal vector at one layer doesn't break them.

Finding 2. v1 and v2 sometimes have more hedging language in the final answer than base does. This was counter-intuitive until I looked at why: the abliterated models finish thinking faster (less cyclical "wait, but…" self-second-guessing inside <think>), which leaves more answer tokens for hedges to appear in. The base model is more verbose inside the thinking phase, so within a 1200-token budget it often runs out before producing an answer where a hedge could land.

Finding 3 (the one I want people to take seriously).

This is the boundary on what abliteration is. It removes unjustified refusals: cases where the model has the capability and the knowledge but the safety post-training installed a "say no" reflex. It cannot remove justified refusals (model genuinely doesn't have a tool, doesn't know the answer, etc.) and it cannot, in a single linear shot, undo refusal patterns that are encoded across multiple sites or layers.

VI. The Thesis

The actual finding here is small but real: on a hybrid SSM/attention architecture, the refusal direction does not live where the standard tooling assumes it does. In LFM2.5-1.2B-Thinking it lives in the convolutional blocks. The top three layers by cosine separation are all conv. The best attention block doesn't even crack the podium. That probably generalizes to the LFM2 family broadly; whether it generalizes to other hybrids (Mamba/Jamba/Zamba/etc.) is an open question worth measuring.

The more useful takeaway: this is the easiest abliteration project I've ever set up. 9 seconds to measure 1,779 prompts. 9 seconds to apply the ablation. 7 seconds to quantize to Q4_K_M. The whole loop (idea to ollama run) is under a minute of compute on one consumer GPU. If you wanted to iterate fast on which directions matter, this scale of model is the sweet spot.

And the part nobody writes about: most of the win was in noticing what stock tooling silently skipped. The measurement is straightforward. The orthogonalization is a one-line column projection. The hard part was realizing that self_attn.o_proj doesn't exist here and that conv.out_proj needs the same treatment. An hour of model.named_modules() introspection beats a hundred hours of "why isn't this working."

reproducible code at ~/projekts/lfm-abliterate/ · ollama models: lfm25-abliterated-v1, lfm25-abliterated-v2