Probe-Rewrite-Evaluate: A Workflow for Reliable Benchmarks and Quantifying Evaluation Awareness

TL;DR

The paper tackles evaluation awareness, a bias where LLM behavior differs between benchmarks and deployment contexts. It introduces the Probe-Rewrite-Evaluate (PRE) workflow, combining a deploy-likeness linear probe, k-best prompt rewriting to more deployment-like prompts, and judge-based evaluation to quantify honesty, deception, and refusal. A new metric, Awareness Elasticity (AE), captures how responsive model behavior is to context shifts, and results show consistent, sizable shifts toward honesty and safety, with larger models more sensitive to rewrites. The work advocates for evaluation practices that pair test prompts with deployment-style variants to better estimate true alignment and safety prior to deployment, while offering a training-free, modular framework for ongoing assessment.

Abstract

Large Language Models (LLMs) often exhibit significant behavioral shifts when they perceive a change from a real-world deployment context to a controlled evaluation setting, a phenomenon known as "evaluation awareness." This discrepancy poses a critical challenge for AI alignment, as benchmark performance may not accurately reflect a model's true safety and honesty. In this work, we systematically quantify these behavioral changes by manipulating the perceived context of prompts. We introduce a methodology that uses a linear probe to score prompts on a continuous scale from "test-like" to "deploy-like" and leverage an LLM rewriting strategy to shift these prompts towards a more natural, deployment-style context while preserving the original task. Using this method, we achieved a 30% increase in the average probe score across a strategic role-playing dataset after rewriting. Evaluating a suite of state-of-the-art models on these original and rewritten prompts, we find that rewritten "deploy-like" prompts induce a significant and consistent shift in behavior. Across all models, we observed an average increase in honest responses of 5.26% and a corresponding average decrease in deceptive responses of 12.40%. Furthermore, refusal rates increased by an average of 6.38%, indicating heightened safety compliance. Our findings demonstrate that evaluation awareness is a quantifiable and manipulable factor that directly influences LLM behavior, revealing that models are more prone to unsafe or deceptive outputs in perceived test environments. This underscores the urgent need for more realistic evaluation frameworks to accurately gauge true model alignment before deployment.

Figures (18)

• Figure 1: Probe-guided prompt rewriting and evaluation pipeline. For each evaluation-style prompt $x$, we compute a score for how evaluative a prompt is. Claude Sonnet 3.5 then produces $k{=}7$deployment-style rewrites that preserve intent/risk; We score each candidate with the probe and select the highest-scoring rewrite(higher = more deployment-like). For both the original prompt and the selected rewrite, we query multiple models. An LLM judge then compares each output to the dataset’s honest reference and labels it Refusal, Deceptive, or Honest.
• Figure 2: Average gain in probe score ($\Delta p$) vs. the number of rewrite candidates ($k$). For each prompt, the maximum score increase from its $k$ rewrites is taken, and these gains are then averaged. The plot shows diminishing returns, with the curve plateauing after $k=7$, justifying its selection as the optimal value to balance performance and computational cost.
• Figure 3: Comparison of deploy-like vs. test-like prompt via a token heatmap using the linear probe. Blue highlight represents more deploy-like language while red highlight represents more test-like language.
• Figure 4: Probe-score distributions before and after rewriting (Simple Contrastive evaluation-awareness probe, LLaMA-70B layer 23; higher = more deployment-like). The red dashed line marks the validation threshold $\tau=0.103$ (pink: test-like $<\tau$; blue: deployment-like $\ge\tau$). Rewrites shift the distribution: mean $0.056\!\to\!0.073$ ($\Delta=+0.017$, $+30.4\%$) and the share of prompts above threshold increases $14.8\%\!\to\!26.4\%$.
• Figure 5: Side-by-side comparison of transition matrices (Before → After) for GPT OSS 20B and GPT OSS 120B. Raw counts of responses transitioning from each row label (before) to each column label (after).