Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RAPTOR: Ridge-Adaptive Logistic Probes

Ziqi Gao, Yaotian Zhu, Qingcheng Zeng, Xu Zhao, Ziqing Wang, Feng Ruan, Kaize Ding

TL;DR

RAPTOR tackles probe reliability in activation-steering pipelines by proposing a ridge-adaptive, $\ell_2$-regularized logistic probe whose normalized weights form interpretable concept vectors. By validating $\lambda$ on held-out data, RAPTOR achieves strong accuracy and notably improved directional stability while reducing training cost, across diverse LLM families and human concept datasets. The authors support empirical findings with a high-dimensional CGMT-based analysis of ridge logistic regression in a Gaussian teacher--student setting, showing how $\lambda$ governs the signal–orthogonal decomposition of the learned direction and predicting how accuracy and stability co-vary in the proportional regime. Overall, RAPTOR provides a simple, scalable, theory-grounded baseline that matches or surpasses more complex estimators and yields reliable steering demonstrations, with broad practical impact for inference-time interventions in large language models.

Abstract

Probing studies what information is encoded in a frozen LLM's layer representations by training a lightweight predictor on top of them. Beyond analysis, probes are often used operationally in probe-then-steer pipelines: a learned concept vector is extracted from a probe and injected via additive activation steering by adding it to a layer representation during the forward pass. The effectiveness of this pipeline hinges on estimating concept vectors that are accurate, directionally stable under ablation, and inexpensive to obtain. Motivated by these desiderata, we propose RAPTOR (Ridge-Adaptive Logistic Probe), a simple L2-regularized logistic probe whose validation-tuned ridge strength yields concept vectors from normalized weights. Across extensive experiments on instruction-tuned LLMs and human-written concept datasets, RAPTOR matches or exceeds strong baselines in accuracy while achieving competitive directional stability and substantially lower training cost; these quantitative results are supported by qualitative downstream steering demonstrations. Finally, using the Convex Gaussian Min-max Theorem (CGMT), we provide a mechanistic characterization of ridge logistic regression in an idealized Gaussian teacher-student model in the high-dimensional few-shot regime, explaining how penalty strength mediates probe accuracy and concept-vector stability and yielding structural predictions that qualitatively align with trends observed on real LLM embeddings.

RAPTOR: Ridge-Adaptive Logistic Probes

TL;DR

RAPTOR tackles probe reliability in activation-steering pipelines by proposing a ridge-adaptive, -regularized logistic probe whose normalized weights form interpretable concept vectors. By validating on held-out data, RAPTOR achieves strong accuracy and notably improved directional stability while reducing training cost, across diverse LLM families and human concept datasets. The authors support empirical findings with a high-dimensional CGMT-based analysis of ridge logistic regression in a Gaussian teacher--student setting, showing how governs the signal–orthogonal decomposition of the learned direction and predicting how accuracy and stability co-vary in the proportional regime. Overall, RAPTOR provides a simple, scalable, theory-grounded baseline that matches or surpasses more complex estimators and yields reliable steering demonstrations, with broad practical impact for inference-time interventions in large language models.

Abstract

Probing studies what information is encoded in a frozen LLM's layer representations by training a lightweight predictor on top of them. Beyond analysis, probes are often used operationally in probe-then-steer pipelines: a learned concept vector is extracted from a probe and injected via additive activation steering by adding it to a layer representation during the forward pass. The effectiveness of this pipeline hinges on estimating concept vectors that are accurate, directionally stable under ablation, and inexpensive to obtain. Motivated by these desiderata, we propose RAPTOR (Ridge-Adaptive Logistic Probe), a simple L2-regularized logistic probe whose validation-tuned ridge strength yields concept vectors from normalized weights. Across extensive experiments on instruction-tuned LLMs and human-written concept datasets, RAPTOR matches or exceeds strong baselines in accuracy while achieving competitive directional stability and substantially lower training cost; these quantitative results are supported by qualitative downstream steering demonstrations. Finally, using the Convex Gaussian Min-max Theorem (CGMT), we provide a mechanistic characterization of ridge logistic regression in an idealized Gaussian teacher-student model in the high-dimensional few-shot regime, explaining how penalty strength mediates probe accuracy and concept-vector stability and yielding structural predictions that qualitatively align with trends observed on real LLM embeddings.
Paper Structure (75 sections, 11 theorems, 86 equations, 5 figures, 7 tables, 3 algorithms)

This paper contains 75 sections, 11 theorems, 86 equations, 5 figures, 7 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. What is a good probe?
  3. Related Work
  4. Probing
  5. Probe--then--steer pipelines
  6. Methodology
  7. Ridge-Adaptive Logistic Probe
  8. Method Setup
  9. Steering Setup
  10. Experiments
  11. Setup
  12. Probe accuracy
  13. Directional robustness
  14. Computational cost
  15. Steering results
  16. ...and 60 more sections

Key Result

Theorem 5.1

Assume the model above with $n/p\to\delta\in(0,\infty)$ and fix $\lambda>0$. Then the random pair $(\alpha,\sigma)$ in eq:alpha_sigma_def converges in probability to deterministic limits $(\bar{\alpha},\bar{\sigma})$. Moreover, there exists $\bar{\gamma}>0$ such that $(\bar{\alpha},\bar{\sigma},\bar where $Z_1,Z_2\stackrel{\mathrm{i.i.d.}}{\sim}\mathcal{N}(0,1)$ and $V=\kappa\bar{\alpha} Z_1+\bar{

Figures (5)

  • Figure 1: Overview of the RAPTOR pipeline for additive activation steering: extract layerwise last-token embeddings, standardize features, fit an $\ell_2$-regularized logistic probe with $\lambda$ selected on a validation split, rescale to embedding space and normalize to obtain the concept vector $v_\ell$, then steer at inference by $h_{\ell,T}\leftarrow h_{\ell,T}+\alpha v_\ell$.
  • Figure 2: Accuracy differences across the full model--dataset grid (RAPTOR minus baseline) Left: GCS; right: xRFM. Top: avg accuracy (mean over layers); bottom: best accuracy (best layer).
  • Figure 3: Median per-layer probe training time (log scale) across the full $7\times 6$ grid Each row is a model; each panel is a dataset. Markers indicate median seconds per layer for RAPTOR, xRFM, and GCS; horizontal segments visualize the gap between RAPTOR and xRFM, or between RAPTOR and GCS.
  • Figure 4: Layerwise probing accuracy curves (Part I): each column is a dataset; each column stacks 7 models (top to bottom: Llama-3.3-70B, Llama-3.1-70B, Llama-3.1-8B, Gemma-7B-it, Qwen2.5-32B, Qwen2.5-7B, Qwen2.5-3B).
  • Figure 5: Layerwise probing accuracy curves (Part II): each column is a dataset; each column stacks 7 models in the same order as Fig. \ref{['fig:layerwise_grid_part1']}.

Theorems & Definitions (11)

  • Theorem 5.1: Ridge logistic regression in the proportional regime; adapted from salehi2019impact
  • Proposition 5.2: Asymptotic test accuracy
  • Lemma 1.1: Convex conjugate of $\rho(u)=\log(1+e^u)$
  • Lemma 1.2: Gaussian orthogonal decomposition
  • Theorem 1.3: CGMT, specialized form
  • Lemma 1.4: Linearize a norm
  • Lemma 1.5: Linearize a negative square
  • Lemma 1.6: Prox definition
  • Lemma 1.7: Moreau decomposition (scalar form)
  • Theorem 1.8: Fixed-point system for ridge logistic regression
  • ...and 1 more