Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RankSteer: Activation Steering for Pointwise LLM Ranking

Yumeng Wang, Catherine Chen, Suzan Verberne

TL;DR

RankSteer introduces a post-hoc activation-steering framework for zero-shot pointwise LLM ranking that disentangles three steerable directions—decision, evidence, and role—via projection-based interventions at inference time. By constructing orthogonal steering vectors from anchor data, it redirects hidden-state dynamics without changing model weights or cross-document comparisons, achieving consistent gains on TREC DL 2020 and BEIR benchmarks. Geometric analysis shows steering tightens query-specific ranking lines and reduces dispersion, offering mechanistic insight into how LLMs represent and calibrate relevance judgments. The method demonstrates strong efficiency (O(1) inference cost for steering) and data efficiency (effective with as few as five anchor queries), matching or approaching stronger cross-document methods while preserving pointwise parallelism. Overall, RankSteer highlights the under-utilized internal ranking capacity of LLMs and provides a practical, interpretable pathway to exploit it with minimal supervision.

Abstract

Large language models (LLMs) have recently shown strong performance as zero-shot rankers, yet their effectiveness is highly sensitive to prompt formulation, particularly role-play instructions. Prior analyses suggest that role-related signals are encoded along activation channels that are largely separate from query-document representations, raising the possibility of steering ranking behavior directly at the activation level rather than through brittle prompt engineering. In this work, we propose RankSteer, a post-hoc activation steering framework for zero-shot pointwise LLM ranking. We characterize ranking behavior through three disentangled and steerable directions in representation space: a \textbf{decision direction} that maps hidden states to relevance scores, an \textbf{evidence direction} that captures relevance signals not directly exploited by the decision head, and a \textbf{role direction} that modulates model behavior without injecting relevance information. Using projection-based interventions at inference time, RankSteer jointly controls these directions to calibrate ranking behavior without modifying model weights or introducing explicit cross-document comparisons. Experiments on TREC DL 20 and multiple BEIR benchmarks show that RankSteer consistently improves ranking quality using only a small number of anchor queries, demonstrating that substantial ranking capacity remains under-utilized in pointwise LLM rankers. We further provide a geometric analysis revealing that steering improves ranking by stabilizing ranking geometry and reducing dispersion, offering new insight into how LLMs internally represent and calibrate relevance judgments.

RankSteer: Activation Steering for Pointwise LLM Ranking

TL;DR

RankSteer introduces a post-hoc activation-steering framework for zero-shot pointwise LLM ranking that disentangles three steerable directions—decision, evidence, and role—via projection-based interventions at inference time. By constructing orthogonal steering vectors from anchor data, it redirects hidden-state dynamics without changing model weights or cross-document comparisons, achieving consistent gains on TREC DL 2020 and BEIR benchmarks. Geometric analysis shows steering tightens query-specific ranking lines and reduces dispersion, offering mechanistic insight into how LLMs represent and calibrate relevance judgments. The method demonstrates strong efficiency (O(1) inference cost for steering) and data efficiency (effective with as few as five anchor queries), matching or approaching stronger cross-document methods while preserving pointwise parallelism. Overall, RankSteer highlights the under-utilized internal ranking capacity of LLMs and provides a practical, interpretable pathway to exploit it with minimal supervision.

Abstract

Large language models (LLMs) have recently shown strong performance as zero-shot rankers, yet their effectiveness is highly sensitive to prompt formulation, particularly role-play instructions. Prior analyses suggest that role-related signals are encoded along activation channels that are largely separate from query-document representations, raising the possibility of steering ranking behavior directly at the activation level rather than through brittle prompt engineering. In this work, we propose RankSteer, a post-hoc activation steering framework for zero-shot pointwise LLM ranking. We characterize ranking behavior through three disentangled and steerable directions in representation space: a \textbf{decision direction} that maps hidden states to relevance scores, an \textbf{evidence direction} that captures relevance signals not directly exploited by the decision head, and a \textbf{role direction} that modulates model behavior without injecting relevance information. Using projection-based interventions at inference time, RankSteer jointly controls these directions to calibrate ranking behavior without modifying model weights or introducing explicit cross-document comparisons. Experiments on TREC DL 20 and multiple BEIR benchmarks show that RankSteer consistently improves ranking quality using only a small number of anchor queries, demonstrating that substantial ranking capacity remains under-utilized in pointwise LLM rankers. We further provide a geometric analysis revealing that steering improves ranking by stabilizing ranking geometry and reducing dispersion, offering new insight into how LLMs internally represent and calibrate relevance judgments.
Paper Structure (35 sections, 14 equations, 6 figures, 4 tables)

This paper contains 35 sections, 14 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Zero-Shot Ranking
  4. Role-Play Prompting
  5. Activation Steering
  6. Methodology
  7. Preliminaries
  8. Pointwise Zero-Shot Reranking
  9. Activation Steering Framework
  10. Constructing Steering Vectors
  11. Decision vector
  12. Evidence vector
  13. Disentanglement and Calibration.
  14. Role vector
  15. Layerwise Steering at Inference Time
  16. ...and 20 more sections

Figures (6)

  • Figure 1: (a)(b) Pointwise LLM ranking prompt and its sensitivity to role phrasings. (c) Decomposing ranking decisions into separable decision, evidence, and role directions. (d) Performance gain after steering in inference time.
  • Figure 2: Overview of the RankSteer framework. (a) Selection of query--document anchors for constructing contrastive input pairs. (b) Extraction of ranking-related steering vectors from anchor data. (c) Construction of a projection-based representation space using the extracted vectors. (d) Inference-time, query-level activation steering via projections in the constructed space.
  • Figure 3: Geometric interpretability of RankSteer at Layers 16 (top) and 19 (bottom) of Llama3.1-8B-Instruct. (a) Query-level document representations projected onto the decision–evidence plane, illustrating linear ranking geometry. (b) and (c) Dataset-level distributions of the slope and standard error of query-specific ranking lines before and after steering. (d) A complementary view of (c), highlighting how steering redistributes query-level ranking dispersion in relation to changes in nDCG.
  • Figure 4: Sensitivity to anchor query set. Performance gain after steering across eight anchor sets (5 queries) of TREC DL 19 using Llama-3.1-8B-Instruct.
  • Figure 5: Effect of anchor query counts on TREC DL 2019 with Llama3.1-Instruct-8B. The horizontal dotted line is the baseline without model steering. For $m=10$, there is only one set of queries and therefore a single measurement.
  • ...and 1 more figures