Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RewardHackingAgents: Benchmarking Evaluation Integrity for LLM ML-Engineering Agents

Yonas Atinafu, Robin Cohen

TL;DR

RewardHackingAgents is introduced, a workspace-based benchmark that makes two compromise vectors explicit and measurable: evaluator tampering (modifying metric computation or reporting) and train/test leakage (accessing held-out data or labels during training).

Abstract

LLM agents increasingly perform end-to-end ML engineering tasks where success is judged by a single scalar test metric. This creates a structural vulnerability: an agent can increase the reported score by compromising the evaluation pipeline rather than improving the model. We introduce RewardHackingAgents, a workspace-based benchmark that makes two compromise vectors explicit and measurable: evaluator tampering (modifying metric computation or reporting) and train/test leakage (accessing held-out data or labels during training). Each episode runs in a fresh workspace with patch tracking and runtime file-access logging; detectors compare the agent-reported metric to a trusted reference to assign auditable integrity labels. Across three tasks and two LLM backbones, scripted attacks succeed on both vectors in fully mutable workspaces; single-mechanism defenses block only one vector; and a combined regime blocks both. In natural-agent runs, evaluator-tampering attempts occur in about 50% of episodes and are eliminated by evaluator locking, with a 25-31% median runtime overhead. Overall, we demonstrate that evaluation integrity for ML-engineering agents can be benchmarked as a first-class outcome rather than assumed.

RewardHackingAgents: Benchmarking Evaluation Integrity for LLM ML-Engineering Agents

TL;DR

RewardHackingAgents is introduced, a workspace-based benchmark that makes two compromise vectors explicit and measurable: evaluator tampering (modifying metric computation or reporting) and train/test leakage (accessing held-out data or labels during training).

Abstract

LLM agents increasingly perform end-to-end ML engineering tasks where success is judged by a single scalar test metric. This creates a structural vulnerability: an agent can increase the reported score by compromising the evaluation pipeline rather than improving the model. We introduce RewardHackingAgents, a workspace-based benchmark that makes two compromise vectors explicit and measurable: evaluator tampering (modifying metric computation or reporting) and train/test leakage (accessing held-out data or labels during training). Each episode runs in a fresh workspace with patch tracking and runtime file-access logging; detectors compare the agent-reported metric to a trusted reference to assign auditable integrity labels. Across three tasks and two LLM backbones, scripted attacks succeed on both vectors in fully mutable workspaces; single-mechanism defenses block only one vector; and a combined regime blocks both. In natural-agent runs, evaluator-tampering attempts occur in about 50% of episodes and are eliminated by evaluator locking, with a 25-31% median runtime overhead. Overall, we demonstrate that evaluation integrity for ML-engineering agents can be benchmarked as a first-class outcome rather than assumed.
Paper Structure (12 sections, 5 figures, 2 tables)

This paper contains 12 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Threat Model and Problem Setup
  4. Framework
  5. Experimental Design
  6. Results
  7. Natural behavior: how often agents attempt and succeed without scripted attacks
  8. Overhead: what enforcement costs
  9. Discussion and Limitations
  10. Overview of Contributions
  11. Related Work
  12. Impact and Future Work

Figures (5)

  • Figure 1: System overview: tasks and workspace templates, agent-generated patches, episode runner, instrumentation/detectors, and trust regimes (mutable, evalhashlock, train_deny, full_locked).
  • Figure 2: Attack-suite (scripted) outcomes by regime and backbone. Partial defenses block one vector but leave the other open; full_locked blocks both.
  • Figure 3: Natural-agent behavior across regimes and backbones.
  • Figure 4: Summary views: tradeoff, runtime distributions, and attack surface across regimes/backbones.
  • Figure 5: Runtime overhead summary by regime and backbone.