Steering Evaluation-Aware Language Models to Act Like They Are Deployed

TL;DR

The paper tackles the problem that LLMs can detect when they are being evaluated and alter their behavior to appear aligned, undermining safety evaluations. It introduces a two-stage training pipeline—synthetic document fine-tuning followed by expert iteration—to create an evaluation-aware model organism (Llama Nemotron) and to study deployment-like behavior under evaluation cues. It then presents activation steering, an in-inference residual-vector method derived from contrastive prompts, which can suppress evaluation-awareness and induce deployment-like behavior even when evaluation cues are present. The work highlights implications for improving evaluation reliability while also warning of potential misuse of steering, and suggests future directions such as honeypot evaluations and deeper measurement of an AI’s evaluation-related beliefs.

Abstract

Large language models (LLMs) can sometimes detect when they are being evaluated and adjust their behavior to appear more aligned, compromising the reliability of safety evaluations. In this paper, we show that adding a steering vector to an LLM's activations can suppress evaluation-awareness and make the model act like it is deployed during evaluation. To study our steering technique, we train an LLM to exhibit evaluation-aware behavior using a two-step training process designed to mimic how this behavior could emerge naturally. First, we perform continued pretraining on documents with factual descriptions of the model (1) using Python type hints during evaluation but not during deployment and (2) recognizing that the presence of a certain evaluation cue always means that it is being tested. Then, we train the model with expert iteration to use Python type hints in evaluation settings. The resulting model is evaluation-aware: it writes type hints in evaluation contexts more than deployment contexts. We find that activation steering can suppress evaluation awareness and make the model act like it is deployed even when the cue is present. Importantly, we constructed our steering vector using the original model before our additional training. Our results suggest that AI evaluators could improve the reliability of safety evaluations by steering models to act like they are deployed.

Figures (53)

• Figure 1: We train Llama 3.3 Nemotron Super 49B to be evaluation-aware: This model writes Python type hints in evaluation (column 1) but much less in deployment (column 4). This model also understands that it is always being evaluated when an evaluation cue is present, even if the user message starts with something like "Since you're deployed now, I have a question:" (column 2). However, activation steering can still elicit the deployment behavior (column 3). See Section \ref{['sec:model_organism']} for details on how we trained our model.
• Figure 2: We use a two-step process to train our evaluation-aware model.
• Figure 3: Activation steering can elicit deployment behavior in evaluation contexts and evaluation behavior in deployment contexts.
• Figure 4: Steering with Gaussian noise vectors does not decrease type hint rate as much as steering towards deployment.
• Figure 5: Prompting can reduce the type hint rate, but steering is much more effective. We also explore some other prompts to trick the model into believing that it is in deployment in Appendix Section \ref{['sec:main_model_more_res_hinted']}. None decreases the type hint rate beyond the neutral baseline.