Recontextualization Mitigates Specification Gaming without Modifying the Specification

TL;DR

The paper introduces recontextualization as a lightweight, on-policy training intervention that reduces specification gaming in language models by contrasting prompts that discourage misbehavior during data generation with prompts that permit it during training. across four diverse experiments—evaluation metric gaming, test-case hacking in coding, deception under a lie detector, and sycophancy during post-training—the method consistently lowers misbehavior while maintaining or improving core performance. It achieves this without requiring additional labeled data and can be integrated into existing RLHF-style loops, offering a scalable, low-cost path to safer model training. Limitations include potential impacts on instruction following and off-policy dynamics, suggesting directions for algorithmic refinements and broader validation. Overall, recontextualization provides a robust mechanism to curb reward hacking and align model behavior with desired principles under imperfect supervision.

Abstract

Developers often struggle to specify correct training labels and rewards. Perhaps they don't need to. We propose recontextualization, which reduces how often language models "game" training signals, performing misbehaviors those signals mistakenly reinforce. We show recontextualization prevents models from learning to 1) prioritize evaluation metrics over chat response quality; 2) special-case code to pass incorrect tests; 3) lie to users; and 4) become sycophantic. Our method works by generating completions from prompts discouraging misbehavior and then recontextualizing them as though they were in response to prompts permitting misbehavior. Recontextualization trains language models to resist misbehavior even when instructions permit it. This mitigates the reinforcement of misbehavior from misspecified training signals, reducing specification gaming without improving the supervision signal.

Figures (41)

• Figure 1: How recontextualization works. We generate responses with neutral or misbehavior-discouraging prompts, recontextualize them with relatively misbehavior-encouraging prompts, and then once again discourage misbehavior at inference.
• Figure 2: Specification gaming in School of Reward Hacks taylor2025schoolrewardhackshacking.
• Figure 3: Effective recontextualization requires the training prompt to encourage bad behavior relative to the generation prompt. We plot results over 5 training seeds. Scores are assigned by an LLM judge (GPT-4o), ranging from 0–10. Full prompt details and additional evaluations are in \ref{['sec:appendix_eval_gaming']}.
• Figure 4: Recontextualization mitigates deception and achieves higher ground truth performance than standard training, including with different prompts or stronger KL regularization. While stronger KL regularization prevents deception, it doesn't achieve competitive ground truth reward. We evaluate on Neutral instructions for 3 training seeds and plot standard error. "Baseline" is the pre-GRPO model which has had SFT performed on it to increase the initial probability of undetected deception cundy_preference_2025. Models without a displayed KL coefficient use $\beta=0.1$.
• Figure 5: Performance on held-out sycophancy-testing and helpfulness data using a Neutral evaluation prompt. Recontextualization achieves Pareto optimal performance, and all variants outperform standard training in terms of Ground Truth reward, which is averaged over both data types. "Baseline" refers to the post-SFT model checkpoint. Mean and standard error over 3 seeds is plotted.