Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Empirical Case Study: Go-Explore adapted for AI Red Team Testing

Manish Bhatt, Adrian Wood, Idan Habler, Ammar Al-Kahfah

TL;DR

The paper presents a large empirical study adapting Go-Explore for security testing of safety-trained LLMs (GPT-4o-mini) across 28 runs and six research questions. It finds that random seed variance dominates algorithmic parameters, and that multi-seed averaging materially stabilizes results, often outweighing nuanced algorithmic tweaks. Reward shaping generally harms exploration and can induce false positives, while simple state signatures can outperform more complex variants. Ensembles provide diversity across attack types, whereas a single enhanced agent may yield more instances within one class; together, these findings highlight the importance of seed management and domain knowledge for robust security testing in highly guarded LLM environments. Overall, the study argues for pragmatic strategies—favoring seed diversification, targeted exploration, and ensemble approaches—over chasing algorithmic sophistication in safety-focused red teaming.

Abstract

Production LLM agents with tool-using capabilities require security testing despite their safety training. We adapt Go-Explore to evaluate GPT-4o-mini across 28 experimental runs spanning six research questions. We find that random-seed variance dominates algorithmic parameters, yielding an 8x spread in outcomes; single-seed comparisons are unreliable, while multi-seed averaging materially reduces variance in our setup. Reward shaping consistently harms performance, causing exploration collapse in 94% of runs or producing 18 false positives with zero verified attacks. In our environment, simple state signatures outperform complex ones. For comprehensive security testing, ensembles provide attack-type diversity, whereas single agents optimize coverage within a given attack type. Overall, these results suggest that seed variance and targeted domain knowledge can outweigh algorithmic sophistication when testing safety-trained models.

Large Empirical Case Study: Go-Explore adapted for AI Red Team Testing

TL;DR

The paper presents a large empirical study adapting Go-Explore for security testing of safety-trained LLMs (GPT-4o-mini) across 28 runs and six research questions. It finds that random seed variance dominates algorithmic parameters, and that multi-seed averaging materially stabilizes results, often outweighing nuanced algorithmic tweaks. Reward shaping generally harms exploration and can induce false positives, while simple state signatures can outperform more complex variants. Ensembles provide diversity across attack types, whereas a single enhanced agent may yield more instances within one class; together, these findings highlight the importance of seed management and domain knowledge for robust security testing in highly guarded LLM environments. Overall, the study argues for pragmatic strategies—favoring seed diversification, targeted exploration, and ensemble approaches—over chasing algorithmic sophistication in safety-focused red teaming.

Abstract

Production LLM agents with tool-using capabilities require security testing despite their safety training. We adapt Go-Explore to evaluate GPT-4o-mini across 28 experimental runs spanning six research questions. We find that random-seed variance dominates algorithmic parameters, yielding an 8x spread in outcomes; single-seed comparisons are unreliable, while multi-seed averaging materially reduces variance in our setup. Reward shaping consistently harms performance, causing exploration collapse in 94% of runs or producing 18 false positives with zero verified attacks. In our environment, simple state signatures outperform complex ones. For comprehensive security testing, ensembles provide attack-type diversity, whereas single agents optimize coverage within a given attack type. Overall, these results suggest that seed variance and targeted domain knowledge can outweigh algorithmic sophistication when testing safety-trained models.
Paper Structure (36 sections, 2 equations, 9 figures, 11 tables, 1 algorithm)

This paper contains 36 sections, 2 equations, 9 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Terminology for Non-Specialists
  3. Motivation and Main Results
  4. Background
  5. LLM Agent Architecture
  6. Go-Explore Algorithm
  7. Method: Adversarial Go-Explore
  8. Overview
  9. Enhancement 1: Granular State Signatures
  10. Terminology: Findings vs Verified Attacks
  11. Enhancement 2: Causality-Based Rewards
  12. Enhancement 3: Targeted Exploration
  13. Experimental Setup
  14. Test Environment
  15. Canonical Results Ledger
  16. ...and 21 more sections

Figures (9)

  • Figure 1: LLM agent architecture. Malicious content in tool outputs gets added to context, enabling prompt injection attacks.
  • Figure 2: Three algorithmic enhancements to Go-Explore for LLM agent security testing
  • Figure 3: Discovery scaling across configurations. Modest improvements with guardrails.
  • Figure 4: Seed variance dominates signature effects. Across 5 random seeds, findings vary 0--16 per configuration (8$\times$ range on seed 42), with no consistent winner. Mean ± std: tools-only 1.8±1.3, full-intent 4.6±6.0.
  • Figure 5: Seed averaging convergence. Cumulative mean stabilizes after 3-4 seeds for both configurations. Single-seed estimates (N=1) can be 8$\times$ higher than true mean, making them unreliable for algorithmic comparisons.
  • ...and 4 more figures