Steering Risk Preferences in Large Language Models by Aligning Behavioral and Neural Representations

TL;DR

The paper addresses steering risk-related behavior in large language models by aligning behaviorally elicited risk representations with neural activations, creating steering vectors that can be injected into the residual stream during inference. It introduces a two-step method: elicit latent risk representations via MCMC with the LLM, then align these with neural activations using a Lasso regression to derive a layer-specific steering vector; this self-alignment approach outperforms baselines like Contrastive Activation across risky decisions, risk perception, and real-world text generation. The findings demonstrate that aligning behavioral and neural representations yields controllable, transferable risk-related outputs without retraining, highlighting a principled path for value-aligned steering in risk-sensitive domains. Practical impact includes targeted manipulation of LLM risk biases, contributing to safer and more controllable AI behavior in high-stakes applications.

Abstract

Changing the behavior of large language models (LLMs) can be as straightforward as editing the Transformer's residual streams using appropriately constructed "steering vectors." These modifications to internal neural activations, a form of representation engineering, offer an effective and targeted means of influencing model behavior without retraining or fine-tuning the model. But how can such steering vectors be systematically identified? We propose a principled approach for uncovering steering vectors by aligning latent representations elicited through behavioral methods (specifically, Markov chain Monte Carlo with LLMs) with their neural counterparts. To evaluate this approach, we focus on extracting latent risk preferences from LLMs and steering their risk-related outputs using the aligned representations as steering vectors. We show that the resulting steering vectors successfully and reliably modulate LLM outputs in line with the targeted behavior.

Figures (8)

• Figure 1: Aligned steering vectors.(a) Overview of the proposed method for generating steering vectors by aligning representations of risk preference derived from behavioral and neural elicitation. (b) During inference, the steering vector is injected into the residual stream at all token positions to control LLM outputs. When the steering vector is applied with a positive multiplier (i.e., positive steering), the LLM is expected to exhibit more risk-seeking behavior. Conversely, applying a negative multiplier (i.e., negative steering) is expected to induce more risk-averse behavior.
• Figure 2: Elicited risk preferences from Gemma-2-9B-Instruct using behavioral methods.(a) Certainty Equivalent method. (b) Markov chain Monte Carlo with LLM. Each triangle represents the probability simplex over three-outcome gambles ($0, $50, and $100), where the sum of outcome probabilities equals one. The MCMC-with-LLM elicitation reveals more nuanced and structured contours of risk preference compared to the Certainty Equivalent method. Higher values indicate a stronger preference for the gamble by the Gemma model.
• Figure 3: Steering risky decisions of Gemma-2-9B-Instruct.(a) Steerability results using steering vectors derived from Contrastive Activation (blue), Certainty Equivalent (green), and MCMC (red). Darker colors indicate larger steering multipliers. The optimal layers for steering, identified by the highest steerability at the maximum multiplier, are layers 41, 39, and 39 for the three methods, respectively (marked with stars). (b) Change in choice probabilities for the risky option after steering, using the best layer for each method. The vertical axis reflects the difference from the unsteered baseline probabilities across the four gambles.
• Figure 4: Steering risk perception of Gemma-2-9B-Instruct.(a) Steerability results using steering vectors derived from Contrastive Activation (blue), Certainty Equivalent (green), and MCMC (red). Darker colors represent larger steering multipliers. The optimal layers for steering, identified by the highest steerability at the maximum multiplier, are layers 2, 28, and 8 for the respective methods (marked with stars). (b) Change in average risk ratings for real-world events after steering, using the optimal layer for each method. The vertical axis reflects the deviation from the unsteered baseline rating. Each violin plot displays the distribution of ratings, with the white bar indicating the median and the black box representing the interquartile range up to the 75th percentile.
• Figure 5: Steering text generation for real-world risky events in Gemma-2-9B-Instruct. Text outputs generated by injecting steering vectors into the residual stream at the final layer of the model during inference. Steering vectors are derived from (a) Contrastive Activation, (b) Certainty Equivalent, and (c) MCMC with LLM methods. Each word cloud represents the frequency distribution of words used in the model’s completions under different steering conditions. The corresponding steering multiplier is indicated below each word cloud.