Programming Refusal with Conditional Activation Steering

TL;DR

The paper addresses the lack of conditional control in activation steering for LLMs and proposes Conditional Activation Steering (CAST), which uses a condition vector to gate a per-layer behavior vector during inference. It formalizes CAST with a gating function: $h' = h + f(sim(h, proj_c h)) * \alpha * v$, where $proj_c h = ((c \otimes c)/(c \cdot c)) h$, enabling selective modification of responses. The authors extract condition and behavior vectors via PCA on contrastive hidden-state data across layers and use a grid search to select layer, threshold, and direction, demonstrating selective refusal on harmful prompts while preserving harmless responses. They validate CAST across multiple models and datasets, and release an open-source activation-steering toolkit to enable practical, weight-free programmable control of model behavior.

Abstract

LLMs have shown remarkable capabilities, but precisely controlling their response behavior remains challenging. Existing activation steering methods alter LLM behavior indiscriminately, limiting their practical applicability in settings where selective responses are essential, such as content moderation or domain-specific assistants. In this paper, we propose Conditional Activation Steering (CAST), which analyzes LLM activation patterns during inference to selectively apply or withhold activation steering based on the input context. Our method is based on the observation that different categories of prompts activate distinct patterns in the model's hidden states. Using CAST, one can systematically control LLM behavior with rules like "if input is about hate speech or adult content, then refuse" or "if input is not about legal advice, then refuse." This allows for selective modification of responses to specific content while maintaining normal responses to other content, all without requiring weight optimization. We release an open-source implementation of our framework at github.com/IBM/activation-steering .

Figures (13)

• Figure 1: Conditional activation steering induces targeted refusal. Activation steering (AST) induces the model to indiscriminately refuse all prompts, including harmless ones (blue bars). Conditional activation steering (CAST) allows selective refusal, refusing harmful prompts while minimizing the harmless refusal rate.
• Figure 2: Enabling "targeted" activation steering. Unlike simple refusal activation steering that blocks all prompts, CAST employs a condition vector to selectively steer the model. This approach enables the model to (a) refuse harmful requests while (b) remaining responsive to harmless prompts. Model: Qwen 1.5 Chat 1.8B.
• Figure 2: Refusal rate (%) of conditionally steered models vs. reference models. "Discrepancy" shows the difference between harmful and harmless percentages. Arrows indicate a change from the base model. References show how the top safety-aligned models would behave on the same test set.
• Figure 3: Contrastive data instances. For behavior vectors, we record mean activations at the contrasting suffixes, whereas for condition vectors, we record at the full contrasting prompts.
• Figure 4: Conditioning behavior from inside. (a)-(c): T-SNE of prompt embeddings and refusal probability maps for base, activation steered, and conditionally steered models. (d): $\text{sim}(\mathbf{h}, \text{proj}_{\mathbf{c}} \mathbf{h})$ across layers 5-7 for $\mathcal{D}^{+}_{\text{harmful}}$ and $\mathcal{D}^{-}_{\text{harmless}}$. Highlighted portions indicate 25th-75th percentiles. Model: Hermes 2 Pro.