Surrogate Fitness Metrics for Interpretable Reinforcement Learning

TL;DR

The paper tackles the interpretability challenge in reinforcement learning by introducing REACT, a framework that generates diverse, informative policy demonstrations through evolutionary optimization of initial states. It defines a joint surrogate fitness combining local diversity, global diversity, and action certainty to select trajectories that reveal edge-case behaviors and decision-making uncertainties, beyond traditional reward-based evaluations. Across discrete gridworlds and continuous FetchReach tasks, REACT improves demonstration fidelity and state-space coverage in early-stage policies and provides qualitative insights even when quantitative fidelity gains are modest for mature systems. The work demonstrates the value of structured, surrogate-based trajectory evaluation for explainability and safety-critical RL, while also showing scenarios where direct fidelity optimization may outperform diversity-driven search, suggesting avenues for adaptive fitness design.

Abstract

We employ an evolutionary optimization framework that perturbs initial states to generate informative and diverse policy demonstrations. A joint surrogate fitness function guides the optimization by combining local diversity, behavioral certainty, and global population diversity. To assess demonstration quality, we apply a set of evaluation metrics, including the reward-based optimality gap, fidelity interquartile means (IQMs), fitness composition analysis, and trajectory visualizations. Hyperparameter sensitivity is also examined to better understand the dynamics of trajectory optimization. Our findings demonstrate that optimizing trajectory selection via surrogate fitness metrics significantly improves interpretability of RL policies in both discrete and continuous environments. In gridworld domains, evaluations reveal significantly enhanced demonstration fidelities compared to random and ablated baselines. In continuous control, the proposed framework offers valuable insights, particularly for early-stage policies, while fidelity-based optimization proves more effective for mature policies. By refining and systematically analyzing surrogate fitness functions, this study advances the interpretability of RL models. The proposed improvements provide deeper insights into RL decision-making, benefiting applications in safety-critical and explainability-focused domains.

Figures (7)

• Figure 1: REACT Architecture
• Figure 2: Comparing encoding lenghts $n \in [4, 5, 6, 7, 8]$ for encoding random 11x11 states
• Figure 3: Comparing individual survival generations over various population sizes.
• Figure 4: Diversity of the individuals of the last generation compared by different operator probabilities
• Figure 5: Evaluation of REACT-generated policy demonstrations in the FlatGrid11\ref{['fig:FlatGrid11']} w.r.t. the fidelity optimization progress \ref{['fig:FlatGrid-Fidelity']}, fitness composition \ref{['fig:FlatGrid-Fitness']}, and final fidelity IQMs and return optimality gaps \ref{['fig:FlatGrid-Ablations']}, comparing REACT with joint fitness (light green), a simple sum (dark green), certainty (orange), local- (light blue), and global diversity (dark blue) fitness to random initial states (red), training initial states (light grey), fidelity-based optimization (dark grey), and visualizations of the states visited by REACT \ref{['fig:REACTHM-FlatGrid']}, Fidelity \ref{['fig:FidelityHM-FlatGrid']} and Random-generated \ref{['fig:RandomHM-FlatGrid']} demonstrations.