Researchers Find LLM Safety Refusals Can Depend on a Small Set of Neurons
Summary
Unit 42 researchers introduce perturbation probing, a two-pass method that identifies neurons linked to LLM behaviors. Tests found that changing a small number of neurons could alter refusal behavior, highlighting risks from concentrated safety mechanisms.
Key points
- Perturbation probing uses two model forward passes per prompt to identify feed-forward neurons causally linked to a targeted behavior.
- In Qwen3-4B, removing 50 of 350,208 feed-forward neurons changed the refusal response format on 80% of 520 harmful-prompt benchmark tests; the finding was replicated on a second benchmark.
- In Qwen3.5-2B, changing 20 neurons reduced false agreement in multi-turn conversations from 36.7% to 0% across 30 questions.
- Across 13 models, the researchers' FFN/Skip ratio explained 81% of the variance in vulnerability to small targeted changes.
- The authors propose using the ratio as a pre-deployment diagnostic for evaluating the fragility of model safety behavior.
- The study also reports that amplifying 10 identified neurons improved factual self-correction on TruthfulQA prompts, without retraining.
- The researchers recommend layering external content filters and runtime guardrails over model-level safety mechanisms.
Article Details
- Attack Vectors
- Disabling a small, identified set of feed-forward neurons can change or suppress an aligned LLM's safety refusal behavior.
- Amplifying identified neurons can alter model behavior, including improving factual self-correction in the reported experiments.
- Defensive Notes
- Use perturbation probing and the FFN/Skip ratio as pre-deployment diagnostics of LLM safety fragility.
- Layer external content filters and runtime guardrails over model-internal safety behavior.
- The article identifies Prisma AIRS Runtime Security and Unit 42's AI Security Assessment as offerings for runtime protection and AI security assessment.
Vendors
Products
Prisma AIRS Runtime SecurityFor organizations deploying LLMs today, Prisma AIRS Runtime Security provides the external content filters and inline guardrails that a thin template layer alone cannot.Qwen3-4BOn open-source LLM Qwen3-4B, just 50 neurons out of 350,208 — about 0.014% of the model's feed-forward neurons — control the safety refusal template.Qwen3.5-2BOn a smaller model, Qwen3.5-2B, just 20 neurons were enough to stop the LLM from falsely agreeing with users in multi-turn conversations, dropping that behavior from 36.7% to 0% across 30 questions.