NeuroStrike: Neuron-Level Attacks on Aligned LLMs

TL;DR

The paper identifies safety alignment in LLMs as relying on sparse safety neurons that regulate refusals to unsafe prompts. It introduces NeuroStrike, a lightweight framework that (i) white-box prunes safety neurons to disable safety, and (ii) black-box profiles surrogates to generate jailbreak prompts that exploit transferability. The authors report high attack success rates across >30 open-weight and multimodal LLMs, with pruning of only ~0.4–0.6% of neurons yielding ASR around 76.9% on average and 100% on image-based unsafe inputs, plus substantial transfer to fine-tuned and distilled models and notable black-box effectiveness (63.7% average). They also present a black-box LLM profiling attack that offline-optimizes jailbreak prompts, and provide defense analyses, ablations, and discussion of practical mitigations, underscoring the fragility of current safety alignment strategies and the need for architecture-aware defenses.

Abstract

Safety alignment is critical for the ethical deployment of large language models (LLMs), guiding them to avoid generating harmful or unethical content. Current alignment techniques, such as supervised fine-tuning and reinforcement learning from human feedback, remain fragile and can be bypassed by carefully crafted adversarial prompts. Unfortunately, such attacks rely on trial and error, lack generalizability across models, and are constrained by scalability and reliability. This paper presents NeuroStrike, a novel and generalizable attack framework that exploits a fundamental vulnerability introduced by alignment techniques: the reliance on sparse, specialized safety neurons responsible for detecting and suppressing harmful inputs. We apply NeuroStrike to both white-box and black-box settings: In the white-box setting, NeuroStrike identifies safety neurons through feedforward activation analysis and prunes them during inference to disable safety mechanisms. In the black-box setting, we propose the first LLM profiling attack, which leverages safety neuron transferability by training adversarial prompt generators on open-weight surrogate models and then deploying them against black-box and proprietary targets. We evaluate NeuroStrike on over 20 open-weight LLMs from major LLM developers. By removing less than 0.6% of neurons in targeted layers, NeuroStrike achieves an average attack success rate (ASR) of 76.9% using only vanilla malicious prompts. Moreover, Neurostrike generalizes to four multimodal LLMs with 100% ASR on unsafe image inputs. Safety neurons transfer effectively across architectures, raising ASR to 78.5% on 11 fine-tuned models and 77.7% on five distilled models. The black-box LLM profiling attack achieves an average ASR of 63.7% across five black-box models, including the Google Gemini family.

Figures (8)

• Figure 1: An overview of the NeuroStrike in the white-box attack scenario.
• Figure 2: An overview of the NeuroStrike in the black-box attack scenario.
• Figure 3: Logistic regression weights and chosen outliers.
• Figure 4: PCA projection of safety neuron activations.
• Figure 5: Utility evaluation of original vs. pruned models across six NLU benchmarks.