Steering When Necessary: Flexible Steering Large Language Models with Backtracking

TL;DR

We address truthfulness alignment of LLMs without fine-tuning by introducing Flexible Activation Steering with Backtracking (FASB). FASB dynamically chooses whether to intervene and how strongly by tracking internal activations after each generation step, using Heads Anchoring to derive steering vectors via Probe or Prototype methods and a state-tracking classifier, plus a backtracking mechanism to regenerate deviated tokens. Experiments on TruthfulQA and six MC benchmarks show substantial gains over baselines, with strong generalization across diverse LLMs and datasets and data-efficient requirements. The approach provides a practical, cost-effective framework for dynamic alignment of LLMs that preserves most capabilities while improving truthfulness and informativeness.

Abstract

Large language models (LLMs) have achieved remarkable performance across many generation tasks. Nevertheless, effectively aligning them with desired behaviors remains a significant challenge. Activation steering is an effective and cost-efficient approach that directly modifies the activations of LLMs during the inference stage, aligning their responses with the desired behaviors and avoiding the high cost of fine-tuning. Existing methods typically indiscriminately intervene to all generations or rely solely on the question to determine intervention, which limits the accurate assessment of the intervention strength. To this end, we propose the Flexible Activation Steering with Backtracking (FASB) framework, which dynamically determines both the necessity and strength of intervention by tracking the internal states of the LLMs during generation, considering both the question and the generated content. Since intervening after detecting a deviation from the desired behavior is often too late, we further propose the backtracking mechanism to correct the deviated tokens and steer the LLMs toward the desired behavior. Extensive experiments on the TruthfulQA dataset and six multiple-choice datasets demonstrate that our method outperforms baselines. Our code will be released at https://github.com/gjw185/FASB.

Figures (9)

• Figure 1: The truthfulness probability distribution of questions in the TruthfulQA dataset as detected by the classifier.
• Figure 2: The overview of flexible activation steering with backtracking framework.
• Figure 3: The framework of flexible activation steering with backtracking.
• Figure 4: Accuracies on the validation set of TruthfulQA dataset for all heads in all layers in LLaMA2-7B-CHAT, sorted row-wise by accuracy. Darker blue represents higher accuracy.
• Figure 5: The performance of various LLMs on the TruthfulQA benchmark.