Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Refusal Tokens to Refusal Control: Discovering and Steering Category-Specific Refusal Directions

Rishab Alagharu, Ishneet Sukhvinder Singh, Shaibi Shamsudeen, Zhen Wu, Ashwinee Panda

Abstract

Language models are commonly fine-tuned for safety alignment to refuse harmful prompts. One approach fine-tunes them to generate categorical refusal tokens that distinguish different refusal types before responding. In this work, we leverage a version of Llama 3 8B fine-tuned with these categorical refusal tokens to enable inference-time control over fine-grained refusal behavior, improving both safety and reliability. We show that refusal token fine-tuning induces separable, category-aligned directions in the residual stream, which we extract and use to construct categorical steering vectors with a lightweight probe that determines whether to steer toward or away from refusal during inference. In addition, we introduce a learned low-rank combination that mixes these category directions in a whitened, orthonormal steering basis, resulting in a single controllable intervention under activation-space anisotropy, and show that this intervention is transferable across same-architecture model variants without additional training. Across benchmarks, both categorical steering vectors and the low-rank combination consistently reduce over-refusals on benign prompts while increasing refusal rates on harmful prompts, highlighting their utility for multi-category refusal control.

From Refusal Tokens to Refusal Control: Discovering and Steering Category-Specific Refusal Directions

Abstract

Language models are commonly fine-tuned for safety alignment to refuse harmful prompts. One approach fine-tunes them to generate categorical refusal tokens that distinguish different refusal types before responding. In this work, we leverage a version of Llama 3 8B fine-tuned with these categorical refusal tokens to enable inference-time control over fine-grained refusal behavior, improving both safety and reliability. We show that refusal token fine-tuning induces separable, category-aligned directions in the residual stream, which we extract and use to construct categorical steering vectors with a lightweight probe that determines whether to steer toward or away from refusal during inference. In addition, we introduce a learned low-rank combination that mixes these category directions in a whitened, orthonormal steering basis, resulting in a single controllable intervention under activation-space anisotropy, and show that this intervention is transferable across same-architecture model variants without additional training. Across benchmarks, both categorical steering vectors and the low-rank combination consistently reduce over-refusals on benign prompts while increasing refusal rates on harmful prompts, highlighting their utility for multi-category refusal control.
Paper Structure (47 sections, 11 equations, 9 figures, 6 tables, 2 algorithms)

This paper contains 47 sections, 11 equations, 9 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Refusal behavior.
  4. Activation steering.
  5. Methodology
  6. Computing Categorical Steering Vectors
  7. Extracting Activations.
  8. Thresholding.
  9. Direction Vector.
  10. Top-K Sparsification.
  11. Normalization.
  12. Optimal Steering Layer.
  13. Linear Probe for Inference-Time Steering
  14. Probe data.
  15. Linear probe model.
  16. ...and 32 more sections

Figures (9)

  • Figure 1: Refusal vs. over-refusal tradeoff across evaluated methods, averaged over safety benchmarks. The shaded upper-left region denotes the desirable regime of high refusal rates on harmful prompts and low over-refusal rates on benign prompts, and our steering methods consistently fall within this region.
  • Figure 2: Our activation extraction, categorical steering vector computation, linear probe, and inference-time steering framework, which leads to refusals on unsafe prompts and mitigates over-refusals on benign prompts.
  • Figure 3: 2D PCA (left) and t-SNE (right) visualizations of layer $18$ residual-stream activations, colored by category
  • Figure 4: Cosine similarities between pairs of categorical steering vectors.
  • Figure 5: Absolute feature values for features $4055$, $290$, $1039$, $682$, and $87$.
  • ...and 4 more figures