Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Steering Language Models Before They Speak: Logit-Level Interventions

Hyeseon An, Shinwoo Park, Hyundong Jin, Yo-Sub Han

TL;DR

Empirical evaluations across three diverse datasets demonstrate that the proposed training-free inference-time logit steering method effectively steers output characteristics, confirming its broad applicability and task-agnostic nature.

Abstract

Steering LLMs is essential for specialized applications such as style-sensitive text rewriting, user-adaptive communication, and toxicity mitigation. Current steering methods, such as prompting-based and activation-based approaches, are widely used to guide model behavior. However, activation-based techniques require deep access to internal layers, while prompting-based steering often fails to provide consistent or fine-grained control. In order to address these limitations, we propose a training-free inference-time logit intervention for controllable generation. Our approach utilizes a statistical token score table derived from z-normalized log-odds of labeled corpora to shift the decoding distribution. Empirical evaluations across three diverse datasets focusing on writing complexity, formality, and toxicity demonstrate that our method effectively steers output characteristics, confirming its broad applicability and task-agnostic nature. Our results show that statistically grounded logit steering can achieve large, consistent, and multi-task control gains: up to +47%p accuracy and 50x f1 improvement.

Steering Language Models Before They Speak: Logit-Level Interventions

TL;DR

Empirical evaluations across three diverse datasets demonstrate that the proposed training-free inference-time logit steering method effectively steers output characteristics, confirming its broad applicability and task-agnostic nature.

Abstract

Steering LLMs is essential for specialized applications such as style-sensitive text rewriting, user-adaptive communication, and toxicity mitigation. Current steering methods, such as prompting-based and activation-based approaches, are widely used to guide model behavior. However, activation-based techniques require deep access to internal layers, while prompting-based steering often fails to provide consistent or fine-grained control. In order to address these limitations, we propose a training-free inference-time logit intervention for controllable generation. Our approach utilizes a statistical token score table derived from z-normalized log-odds of labeled corpora to shift the decoding distribution. Empirical evaluations across three diverse datasets focusing on writing complexity, formality, and toxicity demonstrate that our method effectively steers output characteristics, confirming its broad applicability and task-agnostic nature. Our results show that statistically grounded logit steering can achieve large, consistent, and multi-task control gains: up to +47%p accuracy and 50x f1 improvement.
Paper Structure (37 sections, 12 equations, 3 figures, 5 tables)

This paper contains 37 sections, 12 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Decoding-Time and Prompt-Adjacent Model Steering
  4. Training-Based Steering and Parameter-Efficient Control
  5. Activation-based Steering
  6. Methodology
  7. Preliminaries
  8. Statistical Token Score
  9. Log-odds Ratio Calculation.
  10. Z-score Normalization.
  11. Construction of the Statistical Score Table.
  12. Logit Steering
  13. Candidate Filtering.
  14. Targeted Logit Bias.
  15. Characteristic-guided Sampling.
  16. ...and 22 more sections

Figures (3)

  • Figure 1: Overview of SWAI. It first derives token-level statistical scores from labeled corpora via smoothed one-vs-rest log-odds with variance normalization. During decoding, intervention is confined to a top-$K$ candidate set, where a fixed logit offset is applied to the highest-scoring tokens before sampling from the modified distribution.
  • Figure 2: Class-wise Accuracy on OSE and WikiPol datasets. Reference denotes the Judge LLM performance on the original data before logit steering.
  • Figure 3: Distribution of judge-predicted labels conditioned on the source labels. Each horizontal bar shows, for a fixed source label, the percentage distribution of labels predicted by the judge model after logit steering.