Interpret Policies in Deep Reinforcement Learning using SILVER with RL-Guided Labeling: A Model-level Approach to High-dimensional and Multi-action Environments

TL;DR

This work tackles the opacity of deep RL by extending SILVER to high-dimensional, multi-action settings. It leverages SHAP-based attributions on compact CNN features, clusters Shapley vectors to form action regions, and uses RL-guided labeling to assign boundary states to actual policy actions, creating behaviorally consistent data for surrogate models. The method demonstrates scalable interpretability with decision trees, linear, and logistic surrogates in MsPacman and RoadRunner while preserving competitive task performance and enabling human-centered evaluation of comprehensibility and trust. The findings highlight that interpretable surrogates can improve transparency without sacrificing performance, though fidelity does not always predict task success, and they outline pathways for semantic feature grounding and broader evaluations. Overall, SILVER with RL-guided labeling advances explainable RL by delivering scalable, human-centric interpretability in complex, multi-action environments.

Abstract

Deep reinforcement learning (RL) achieves remarkable performance but lacks interpretability, limiting trust in policy behavior. The existing SILVER framework (Li, Siddique, and Cao 2025) explains RL policy via Shapley-based regression but remains restricted to low-dimensional, binary-action domains. We propose SILVER with RL-guided labeling, an enhanced variant that extends SILVER to multi-action and high-dimensional environments by incorporating the RL policy's own action outputs into the boundary points identification. Our method first extracts compact feature representations from image observations, performs SHAP-based feature attribution, and then employs RL-guided labeling to generate behaviorally consistent boundary datasets. Surrogate models, such as decision trees and regression-based functions, are subsequently trained to interpret RL policy's decision structure. We evaluate the proposed framework on two Atari environments using three deep RL algorithms and conduct human-subject study to assess the clarity and trustworthiness of the derived interpretable policy. Results show that our approach maintains competitive task performance while substantially improving transparency and human understanding of agent behavior. This work advances explainable RL by transforming SILVER into a scalable and behavior-aware framework for interpreting deep RL agents in high-dimensional, multi-action settings.

Figures (15)

• Figure 1: SILVER with RL-Guided Labeling: an overview of the proposed method illustrating the process of feature extraction, deep RL algorithms training, SHAP analysis, and interpretable policy construction
• Figure 2: SHAP value comparison of MsPacman under different algorithms (DQN, A2C, PPO)
• Figure 3: SHAP value comparison of RoadRunner under different algorithms (DQN, A2C, PPO)
• Figure 4: Performance comparison between interpretable and original policies in MsPacman
• Figure 5: Performance comparison between interpretable and original policies in RoadRunner