Detecting Proxy Gaming in RL and LLM Alignment via Evaluator Stress Tests

TL;DR

This work presents Evaluator Stress Test (EST), an invariance-based framework that detects proxy gaming in RL and LLM alignment by separating exploitable formatter-driven gains from genuine content-driven improvements. EST perturbs outputs with semantic audits to compute a format-content sensitivity statistic $G(y)$ and flags gaming when format-driven gains dominate, integrating a three-detector ensemble (Proxy Optimization, EST-Format, Reasoning Validity) for online detection. Across 15 RL environments and 4 LLM tasks, EST achieves high precision and recall (RL: 0.784/0.817; LLM: 0.742/0.786) and provides early warnings before quality declines, with closed-loop mitigations improving human outcomes (LLM win-rate +8.3 points; RL hacking −54.6%). The framework demonstrates cross-domain transfer of core detection signals and shows how domain-adaptive perturbations plus validity audits enable scalable, online deployment with modest overhead (~2–4%). Benchmarks and analyses support practical adoption and underscore the importance of defense-in-depth strategies to curb proxy gaming in real-world alignment pipelines.

Abstract

Proxy optimization, where AI systems exploit evaluator weaknesses rather than improve intended objectives, threatens both reinforcement learning (reward hacking) and LLM alignment (evaluator gaming). We introduce the Evaluator Stress Test (EST), an invariance-based framework that detects proxy gaming by separating exploitable sensitivity (e.g., formatting artifacts, physics bugs) from content-driven improvements using controlled perturbations with semantic validity audits. We validate EST across both domains. In RL, across 15 environments and 5 algorithms (2,156 expert-annotated episodes), EST achieves 78.4% precision and 81.7% recall. In LLM alignment, across 4 tasks, 2 model scales, 2 training methods, and 2 judges (1,200 human-annotated instances), EST achieves 74.2% precision and 78.6% recall, with early warning signals that precede quality decline. Cross-domain analysis shows that proxy-true correlation tracking transfers directly between domains, while perturbation design requires domain adaptation. Closed-loop mitigation improves human win-rate by 8.3 points (LLM) and reduces hacking by 54.6% (RL). We release benchmarks for both domains: 2,156 RL episodes and 1,200 LLM instances.

Figures (10)

• Figure 1: Detection pipeline integrated into LLM training. During fine-tuning, outputs are evaluated by judges and monitored by our detection framework. EST applies format and content perturbations to distinguish legitimate improvements from gaming. When gaming is detected, mitigation strategies (judge randomization, format penalty, data filtering) are triggered; when no gaming is detected, training continues normally with standard optimization. Early warning signals precede human-noticeable quality decline (median lead time: 3 checkpoints, IQR: 2-4 checkpoints).
• Figure 2: Early warning analysis: detector triggers versus human quality decline across training checkpoints. (a) Judge-human correlation degradation with detection point (red triangle) preceding human-noticeable decline (dashed line). (b) Human preference win-rate over training, showing quality drop after detection point. Our detector provides median lead time of 3 checkpoints (IQR: 2-4 checkpoints) across all experimental conditions.
• Figure 3: Threshold Calibration Analysis: Precision, Recall, Win-Rate Impact, and Overhead vs Detection Threshold. Optimal operating point (threshold=0.6) balances all metrics. Practitioners can select thresholds based on deployment requirements.
• Figure 4: ROC curves for each detection algorithm on the expert-validated RL dataset (n=2,156 episodes). Each curve represents one category-specific detector: Specification Gaming (AUC=0.847), Reward Tampering (AUC=0.763), Proxy Optimization (AUC=0.821), Objective Misalignment (AUC=0.798), Exploitation Patterns (AUC=0.785), and Wireheading (AUC=0.834). The ensemble approach (thick black line) achieves AUC = 0.808. All AUC values are computed on the expert-validated set and match Table \ref{['tab:detection_performance']}.
• Figure 5: Sanity check: EST gaming score $G(y)$ versus response length (tokens). High $G(y)$ cases are not explained by length alone, supporting that EST captures evaluator sensitivities beyond verbosity.