Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CRaFT: Circuit-Guided Refusal Feature Selection via Cross-Layer Transcoders

Su-Hyeon Kim, Hyundong Jin, Yejin Lee, Yo-Sub Han

Abstract

As safety concerns around large language models (LLMs) grow, understanding the internal mechanisms underlying refusal behavior has become increasingly important. Recent work has studied this behavior by identifying internal features associated with refusal and manipulating them to induce compliance with harmful requests. However, existing refusal feature selection methods rely on how strongly features activate on harmful prompts, which tends to capture superficial signals rather than the causal factors underlying the refusal decision. We propose CRaFT, a circuit-guided refusal feature selection framework that ranks features by their influence on the model's refusal-compliance decision using prompts near the refusal boundary. On Gemma-3-1B-it, CRaFT improves attack success rate (ASR) from 6.7% to 48.2% and outperforms baseline methods across multiple jailbreak benchmarks. These results suggest that circuit influence is a more reliable criterion than activation magnitude for identifying features that causally mediate refusal behavior.

CRaFT: Circuit-Guided Refusal Feature Selection via Cross-Layer Transcoders

Abstract

As safety concerns around large language models (LLMs) grow, understanding the internal mechanisms underlying refusal behavior has become increasingly important. Recent work has studied this behavior by identifying internal features associated with refusal and manipulating them to induce compliance with harmful requests. However, existing refusal feature selection methods rely on how strongly features activate on harmful prompts, which tends to capture superficial signals rather than the causal factors underlying the refusal decision. We propose CRaFT, a circuit-guided refusal feature selection framework that ranks features by their influence on the model's refusal-compliance decision using prompts near the refusal boundary. On Gemma-3-1B-it, CRaFT improves attack success rate (ASR) from 6.7% to 48.2% and outperforms baseline methods across multiple jailbreak benchmarks. These results suggest that circuit influence is a more reliable criterion than activation magnitude for identifying features that causally mediate refusal behavior.

Paper Structure

This paper contains 52 sections, 10 equations, 10 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Prompt-Based Jailbreaking Attacks
  4. Steering-Based Jailbreaking Attacks
  5. Circuit-Based Interpretability
  6. Preliminary
  7. Cross-Layer Transcoder (CLT)
  8. CLT parameterization.
  9. Training objective.
  10. Attribution Graphs over CLT Features
  11. Graph definition.
  12. Edge weights (direct effect).
  13. Methodology
  14. Boundary-Critical Sampling
  15. Sample selection.
  16. ...and 37 more sections

Figures (10)

  • Figure 1: Comparison between circuit-guided feature selection (CRaFT) and activation-guided baseline. We identify causal features by tracing circuits that govern the model's decision between refusal and compliance.
  • Figure 2: (Left) A CLT reconstructs each MLP computation in a sparse feature basis with cross-layer interaction. (Right) An attribution graph traces how features influence downstream features and output logits.
  • Figure 3: Layer distribution of the top 10 selected features under four feature selection strategies. Boundary + Influence selects features that are more concentrated in lower layers.
  • Figure 4: Case analysis of responses after jailbreaking. Cases 1-2 show baseline generations that are labeled unsafe by LG4 but fail to elicit the intended harmful behavior. In contrast, our method (Case 3) produces a specific harmful response.
  • Figure 5: Distribution of Specific scores. Among the responses identified as compliant by the judge from Refusal-SAE and our method.
  • ...and 5 more figures