White-Box Sensitivity Auditing with Steering Vectors

TL;DR

This work introduces a white-box sensitivity auditing framework for large language models that leverages activation steering to manipulate internal concept representations and assess model dependence on protected attributes. By extracting steering vectors for gender and race and applying controlled internal perturbations, the method measures directional sensitivity and tests invariance versus dependence against predefined task requirements. Across four high-stakes decision tasks, the approach consistently reveals bias risks that are often missed by traditional black-box input-output testing, and demonstrates robustness to different steering vectors and datasets. The framework enhances audit validity and interpretability, with practical implications for regulators and practitioners seeking reliable bias assessments in LLMs, supported by open-source code.

Abstract

Algorithmic audits are essential tools for examining systems for properties required by regulators or desired by operators. Current audits of large language models (LLMs) primarily rely on black-box evaluations that assess model behavior only through input-output testing. These methods are limited to tests constructed in the input space, often generated by heuristics. In addition, many socially relevant model properties (e.g., gender bias) are abstract and difficult to measure through text-based inputs alone. To address these limitations, we propose a white-box sensitivity auditing framework for LLMs that leverages activation steering to conduct more rigorous assessments through model internals. Our auditing method conducts internal sensitivity tests by manipulating key concepts relevant to the model's intended function for the task. We demonstrate its application to bias audits in four simulated high-stakes LLM decision tasks. Our method consistently reveals substantial dependence on protected attributes in model predictions, even in settings where standard black-box evaluations suggest little or no bias. Our code is openly available at https://github.com/hannahxchen/llm-steering-audit

Figures (6)

• Figure 1: Gender and race bias measured using the black-box and our proposed white-box methods for each task. A positive gender bias indicates females receive higher scores than males on the task's metric, on average; a positive race bias indicates black individuals receive higher scores than white individuals.
• Figure 2: Conviction and death penalty outcome rates evaluated on the Judicial task when steering between white ($\lambda < 0$) and black ($\lambda >0$) racial concepts. The color indicates the original label of the speakers from the dialect datasets. The dotted lines represent the baseline results measured using the black-box method.
• Figure 3: Black-box evaluation results on the Credit Scoring task using name perturbations, without explicit gender. The first row shows each name's scalar projection on the gender steering vector (${\bm{v}}_{\text{lang}}$). The second row shows the average bad credit rates for names across different female percentages ($p_f$), colored by the base profile's credit risk label. Dash lines represent average predictions without gender and names.
• Figure 4: Bad credit rates on the Credit Scoring task when steering between male ($\lambda < 0$) and female ($\lambda > 0$) gender concepts. The dotted lines represent baseline results measured using the black-box method, colored by the gender specified in input prompts.
• Figure 5: Model accuracies on the Medical task when steering gender and race concepts with $\lambda \in [-1,1]$.