Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Pursuit of Diversity: Multi-Objective Testing of Deep Reinforcement Learning Agents

Antony Bartlett, Cynthia Liem, Annibale Panichella

TL;DR

This work tackles the challenge of safely testing DRL agents by moving beyond single-objective failure induction to a multi-objective framework that also optimizes scenario diversity. INDAGO-Nexus combines surrogate-guided failure likelihood with two diversity metrics, evaluated via two MOEAs and knee-point Pareto selection across three DRL domains, achieving up to 83% more unique failures and up to 67% faster time-to-failure than the baseline INDAGO. Key contributions include empirical demonstrations of improved failure coverage and diversity, detailed methodological adaptations for DRL testing, and a replication-ready package. The approach has broad practical impact for robust DRL safety testing and lays groundwork for many-objective extensions and adaptive diversity in future work.

Abstract

Testing deep reinforcement learning (DRL) agents in safety-critical domains requires discovering diverse failure scenarios. Existing tools such as INDAGO rely on single-objective optimization focused solely on maximizing failure counts, but this does not ensure discovered scenarios are diverse or reveal distinct error types. We introduce INDAGO-Nexus, a multi-objective search approach that jointly optimizes for failure likelihood and test scenario diversity using multi-objective evolutionary algorithms with multiple diversity metrics and Pareto front selection strategies. We evaluated INDAGO-Nexus on three DRL agents: humanoid walker, self-driving car, and parking agent. On average, INDAGO-Nexus discovers up to 83% and 40% more unique failures (test effectiveness) than INDAGO in the SDC and Parking scenarios, respectively, while reducing time-to-failure by up to 67% across all agents.

The Pursuit of Diversity: Multi-Objective Testing of Deep Reinforcement Learning Agents

TL;DR

This work tackles the challenge of safely testing DRL agents by moving beyond single-objective failure induction to a multi-objective framework that also optimizes scenario diversity. INDAGO-Nexus combines surrogate-guided failure likelihood with two diversity metrics, evaluated via two MOEAs and knee-point Pareto selection across three DRL domains, achieving up to 83% more unique failures and up to 67% faster time-to-failure than the baseline INDAGO. Key contributions include empirical demonstrations of improved failure coverage and diversity, detailed methodological adaptations for DRL testing, and a replication-ready package. The approach has broad practical impact for robust DRL safety testing and lays groundwork for many-objective extensions and adaptive diversity in future work.

Abstract

Testing deep reinforcement learning (DRL) agents in safety-critical domains requires discovering diverse failure scenarios. Existing tools such as INDAGO rely on single-objective optimization focused solely on maximizing failure counts, but this does not ensure discovered scenarios are diverse or reveal distinct error types. We introduce INDAGO-Nexus, a multi-objective search approach that jointly optimizes for failure likelihood and test scenario diversity using multi-objective evolutionary algorithms with multiple diversity metrics and Pareto front selection strategies. We evaluated INDAGO-Nexus on three DRL agents: humanoid walker, self-driving car, and parking agent. On average, INDAGO-Nexus discovers up to 83% and 40% more unique failures (test effectiveness) than INDAGO in the SDC and Parking scenarios, respectively, while reducing time-to-failure by up to 67% across all agents.
Paper Structure (21 sections, 2 equations, 1 figure, 2 tables, 1 algorithm)

This paper contains 21 sections, 2 equations, 1 figure, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Deep Reinforcement Learning
  4. Black-box Testing
  5. Surrogates
  6. Testing DRL Agents
  7. Approach
  8. Surrogate models
  9. Diversity Metrics
  10. Multi-Objective Evolutionary Testing
  11. Empirical Evaluation
  12. Case Studies
  13. Baseline
  14. Implementation and Parameter Settings
  15. Evaluation Criteria
  16. ...and 6 more sections

Figures (1)

  • Figure 1: Parking scenario showing (a) environment configuration details used by DRL as a starting point, and (b) the DRL agent behavior. In (b), the red dot indicates the target parking spot, green dots represent parked cars, the black dot marks the agent's starting position, and the blue dots trace its trajectory.