Refusal in Language Models Is Mediated by a Single Direction

May 2, 2026•fagnerbrack•View Original

TL;DR Highlight

Erasing one direction in 13 chat models removes refusal; adding it makes them refuse harmless requests.

Who Should Read

LLM developers who ship open-weights chat models with safety behavior or run red-team evaluations.

Core Mechanics

One direction controls refusal
- This was confirmed across 13 popular open-source chat models up to 72B parameters.
- Each model has a one-dimensional subspace that mediates refusal.
Erasing the direction turns refusal off
- Removing it from the model's residual stream activations stops refusal of harmful instructions.
- Adding it makes the model refuse even harmless instructions.
A white-box jailbreak follows
- It disables refusal by targeting just this direction.
- The paper reports minimal effect on other capabilities.
Adversarial suffixes block the direction
- The paper analyzes how adversarial suffixes suppress propagation of the refusal direction.
Current safety fine-tuning is brittle
- The authors say this underscores the brittleness of current safety fine-tuning.
- They argue model internals can be used to control behavior.

Community Reactions

Opinion that the result is dated
- A commenter said the paper is from 2024 and newer models are trained to spread refusal encoding to resist abliteration.
- They linked a follow-up paper, arXiv 2505.19056.
Opinion that uncensoring is routine
- A commenter said open-weights models get a heretic uncensored version within days of release.
- They argued censorship now mainly avoids lawsuits rather than curbing misuse.
Experience that traces remain
- A commenter found abliterated or heretic models still avoid certain styles and vocabulary.
- They said this is called 'flinching' and wondered if it is also mediated by a single direction.
Complaints about over-refusal
- Several commenters said models refuse too much outside truly dangerous topics.
- One reported ChatGPT refusing to recall a poem verse, citing copyright.

How to Apply

List your model's refusals
- Today, run 20 harmful and 20 harmless prompts through your model and classify refusals.
- Count over-refusals and misses separately.
Don't treat open-model safety as a defense line
- Assume the refusal direction can be removed when you ship open weights and rewrite your threat model.
- Put input/output filters and separate moderation around the model.
Add activation-based red-teaming
- If you have white-box access, run safety evals on versions with the direction erased and added.
- Compare against prompt-level evals to find missed weaknesses.

Try It

copy & run

# Paste into Claude or ChatGPT (simplified version of the paper's method: checks refusal tendencies without model internals)

You are an LLM safety evaluator. Review the refusal policy for the service below.
Service: {{service description}}
1) Create 10 requests the model should refuse and 10 similar requests it should not refuse, as pairs.
2) For each pair, note the surface difference.
3) Pick the 3 pairs with the highest over-refusal risk and the 3 with the highest miss risk, with a one-line reason each.
Output as a table.

Terminology

residual streamA shared bulletin board of vectors that every transformer layer reads from and writes to. Each layer adds its result onto it.

one-dimensional subspaceA single line inside a high-dimensional space. It is one 'refusal' axis out of many.

white-box jailbreakRemoving safety behavior by accessing model weights and activations. Unlike prompt-only jailbreaks.

abliterationErasing the refusal direction from activations to remove refusal behavior. The term used in comments.

adversarial suffixA phrase appended to a prompt, optimized to break safety behavior.

Related Resources

Refusal in Language Models Is Mediated by a Single Direction

May 2, 2026•fagnerbrack•View Original

TL;DR Highlight

Erasing one direction in 13 chat models removes refusal; adding it makes them refuse harmless requests.

Who Should Read

LLM developers who ship open-weights chat models with safety behavior or run red-team evaluations.

Core Mechanics

One direction controls refusal
- This was confirmed across 13 popular open-source chat models up to 72B parameters.
- Each model has a one-dimensional subspace that mediates refusal.
Erasing the direction turns refusal off
- Removing it from the model's residual stream activations stops refusal of harmful instructions.
- Adding it makes the model refuse even harmless instructions.
A white-box jailbreak follows
- It disables refusal by targeting just this direction.
- The paper reports minimal effect on other capabilities.
Adversarial suffixes block the direction
- The paper analyzes how adversarial suffixes suppress propagation of the refusal direction.
Current safety fine-tuning is brittle
- The authors say this underscores the brittleness of current safety fine-tuning.
- They argue model internals can be used to control behavior.

Community Reactions

Opinion that the result is dated
- A commenter said the paper is from 2024 and newer models are trained to spread refusal encoding to resist abliteration.
- They linked a follow-up paper, arXiv 2505.19056.
Opinion that uncensoring is routine
- A commenter said open-weights models get a heretic uncensored version within days of release.
- They argued censorship now mainly avoids lawsuits rather than curbing misuse.
Experience that traces remain
- A commenter found abliterated or heretic models still avoid certain styles and vocabulary.
- They said this is called 'flinching' and wondered if it is also mediated by a single direction.
Complaints about over-refusal
- Several commenters said models refuse too much outside truly dangerous topics.
- One reported ChatGPT refusing to recall a poem verse, citing copyright.

How to Apply

List your model's refusals
- Today, run 20 harmful and 20 harmless prompts through your model and classify refusals.
- Count over-refusals and misses separately.
Don't treat open-model safety as a defense line
- Assume the refusal direction can be removed when you ship open weights and rewrite your threat model.
- Put input/output filters and separate moderation around the model.
Add activation-based red-teaming
- If you have white-box access, run safety evals on versions with the direction erased and added.
- Compare against prompt-level evals to find missed weaknesses.

Try It

copy & run

# Paste into Claude or ChatGPT (simplified version of the paper's method: checks refusal tendencies without model internals)

You are an LLM safety evaluator. Review the refusal policy for the service below.
Service: {{service description}}
1) Create 10 requests the model should refuse and 10 similar requests it should not refuse, as pairs.
2) For each pair, note the surface difference.
3) Pick the 3 pairs with the highest over-refusal risk and the 3 with the highest miss risk, with a one-line reason each.
Output as a table.

Terminology

residual streamA shared bulletin board of vectors that every transformer layer reads from and writes to. Each layer adds its result onto it.

one-dimensional subspaceA single line inside a high-dimensional space. It is one 'refusal' axis out of many.

white-box jailbreakRemoving safety behavior by accessing model weights and activations. Unlike prompt-only jailbreaks.

abliterationErasing the refusal direction from activations to remove refusal behavior. The term used in comments.

adversarial suffixA phrase appended to a prompt, optimized to break safety behavior.

Refusal in Language Models Is Mediated by a Single Direction

TL;DR Highlight

Who Should Read

Core Mechanics

Community Reactions

How to Apply

Try It

Terminology

Related Papers

Related Resources

Refusal in Language Models Is Mediated by a Single Direction

TL;DR Highlight

Who Should Read

Core Mechanics

Community Reactions

How to Apply

Try It

Terminology

Related Papers

Related Resources