Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Interpretable Learning Dynamics in Unsupervised Reinforcement Learning

Shashwat Pandey

TL;DR

This work addresses the interpretability gap in unsupervised reinforcement learning by introducing a unified framework that combines Grad-CAM, Layer-wise Relevance Propagation, and VAE-based representation analysis to study attention and abstraction. By evaluating five agents—DQN, ICM, RND, PPO, and Transformer-RND—on CoinRun, the authors show that curiosity-driven agents develop broader and more dynamic attention and richer latent structures, with Transformer-RND exhibiting the strongest combination of attention breadth and compact latent representations. The findings highlight how architectural inductive biases and training signals shape internal agent dynamics, offering diagnostic tools beyond reward-based metrics to diagnose perception, exploration, and abstraction. This framework has practical significance for building interpretable, robust, and generalizable RL systems in real-world domains like robotics and autonomous systems.

Abstract

We present an interpretability framework for unsupervised reinforcement learning (URL) agents, aimed at understanding how intrinsic motivation shapes attention, behavior, and representation learning. We analyze five agents DQN, RND, ICM, PPO, and a Transformer-RND variant trained on procedurally generated environments, using Grad-CAM, Layer-wise Relevance Propagation (LRP), exploration metrics, and latent space clustering. To capture how agents perceive and adapt over time, we introduce two metrics: attention diversity, which measures the spatial breadth of focus, and attention change rate, which quantifies temporal shifts in attention. Our findings show that curiosity-driven agents display broader, more dynamic attention and exploratory behavior than their extrinsically motivated counterparts. Among them, TransformerRND combines wide attention, high exploration coverage, and compact, structured latent representations. Our results highlight the influence of architectural inductive biases and training signals on internal agent dynamics. Beyond reward-centric evaluation, the proposed framework offers diagnostic tools to probe perception and abstraction in RL agents, enabling more interpretable and generalizable behavior.

Interpretable Learning Dynamics in Unsupervised Reinforcement Learning

TL;DR

This work addresses the interpretability gap in unsupervised reinforcement learning by introducing a unified framework that combines Grad-CAM, Layer-wise Relevance Propagation, and VAE-based representation analysis to study attention and abstraction. By evaluating five agents—DQN, ICM, RND, PPO, and Transformer-RND—on CoinRun, the authors show that curiosity-driven agents develop broader and more dynamic attention and richer latent structures, with Transformer-RND exhibiting the strongest combination of attention breadth and compact latent representations. The findings highlight how architectural inductive biases and training signals shape internal agent dynamics, offering diagnostic tools beyond reward-based metrics to diagnose perception, exploration, and abstraction. This framework has practical significance for building interpretable, robust, and generalizable RL systems in real-world domains like robotics and autonomous systems.

Abstract

We present an interpretability framework for unsupervised reinforcement learning (URL) agents, aimed at understanding how intrinsic motivation shapes attention, behavior, and representation learning. We analyze five agents DQN, RND, ICM, PPO, and a Transformer-RND variant trained on procedurally generated environments, using Grad-CAM, Layer-wise Relevance Propagation (LRP), exploration metrics, and latent space clustering. To capture how agents perceive and adapt over time, we introduce two metrics: attention diversity, which measures the spatial breadth of focus, and attention change rate, which quantifies temporal shifts in attention. Our findings show that curiosity-driven agents display broader, more dynamic attention and exploratory behavior than their extrinsically motivated counterparts. Among them, TransformerRND combines wide attention, high exploration coverage, and compact, structured latent representations. Our results highlight the influence of architectural inductive biases and training signals on internal agent dynamics. Beyond reward-centric evaluation, the proposed framework offers diagnostic tools to probe perception and abstraction in RL agents, enabling more interpretable and generalizable behavior.
Paper Structure (27 sections, 4 equations, 11 figures, 1 table)

This paper contains 27 sections, 4 equations, 11 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Agents and Environment
  4. Training Configuration
  5. Intrinsic Motivation Formulations
  6. ICM.
  7. RND.
  8. Transformer-based Policy Model
  9. Interpretability Framework
  10. VAE-based Representation Analysis
  11. Experiments and Results
  12. Training Performance
  13. Visual Attention Evolution
  14. Representation Quality via VAE
  15. Latent Space Visualization
  16. ...and 12 more sections

Figures (11)

  • Figure 1: Reward curves showing intrinsic, extrinsic, and episode returns for different agent groups.
  • Figure 2: Grad-CAM visualizations over time for ICM (top) vs. DQN (bottom). ICM focuses on actionable areas, DQN remains noisy.
  • Figure 3: VAE reconstructions for all agents. Each block shows original observations (top) and reconstructions (bottom). ICM and Transformer-RND preserve structural detail, while DQN fails to capture coherent semantics.
  • Figure 4: UMAP projection comparison. Transformer-RND (left) forms structured, clustered representations; DQN (right) shows disorganized, overlapping embeddings.
  • Figure 5: DQN: Grad-CAM and LRP visualizations across training steps.
  • ...and 6 more figures