Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Using Part-based Representations for Explainable Deep Reinforcement Learning

Manos Kirtas, Konstantinos Tsampazis, Loukia Avramelou, Nikolaos Passalis, Anastasios Tefas

TL;DR

The paper tackles the lack of transparency in deep reinforcement learning by enabling part-based, interpretable representations through non-negative actor parameters. It introduces a non-negative training pipeline for PPO that uses an exponential initialization for the actor and a sign-preserving update to maintain non-negativity, paired with standard SGD for the critic. Empirical results on CartPole in a high-fidelity 3D robotics setting show higher, more stable rewards and provide visual evidence of interpretable part-based weights. The work suggests potential extensions to value-based methods like DQN and to distillation-based guidance to further improve explainability and robustness.

Abstract

Utilizing deep learning models to learn part-based representations holds significant potential for interpretable-by-design approaches, as these models incorporate latent causes obtained from feature representations through simple addition. However, training a part-based learning model presents challenges, particularly in enforcing non-negative constraints on the model's parameters, which can result in training difficulties such as instability and convergence issues. Moreover, applying such approaches in Deep Reinforcement Learning (RL) is even more demanding due to the inherent instabilities that impact many optimization methods. In this paper, we propose a non-negative training approach for actor models in RL, enabling the extraction of part-based representations that enhance interpretability while adhering to non-negative constraints. To this end, we employ a non-negative initialization technique, as well as a modified sign-preserving training method, which can ensure better gradient flow compared to existing approaches. We demonstrate the effectiveness of the proposed approach using the well-known Cartpole benchmark.

Using Part-based Representations for Explainable Deep Reinforcement Learning

TL;DR

The paper tackles the lack of transparency in deep reinforcement learning by enabling part-based, interpretable representations through non-negative actor parameters. It introduces a non-negative training pipeline for PPO that uses an exponential initialization for the actor and a sign-preserving update to maintain non-negativity, paired with standard SGD for the critic. Empirical results on CartPole in a high-fidelity 3D robotics setting show higher, more stable rewards and provide visual evidence of interpretable part-based weights. The work suggests potential extensions to value-based methods like DQN and to distillation-based guidance to further improve explainability and robustness.

Abstract

Utilizing deep learning models to learn part-based representations holds significant potential for interpretable-by-design approaches, as these models incorporate latent causes obtained from feature representations through simple addition. However, training a part-based learning model presents challenges, particularly in enforcing non-negative constraints on the model's parameters, which can result in training difficulties such as instability and convergence issues. Moreover, applying such approaches in Deep Reinforcement Learning (RL) is even more demanding due to the inherent instabilities that impact many optimization methods. In this paper, we propose a non-negative training approach for actor models in RL, enabling the extraction of part-based representations that enhance interpretability while adhering to non-negative constraints. To this end, we employ a non-negative initialization technique, as well as a modified sign-preserving training method, which can ensure better gradient flow compared to existing approaches. We demonstrate the effectiveness of the proposed approach using the well-known Cartpole benchmark.
Paper Structure (4 sections, 11 equations, 3 figures, 1 table, 1 algorithm)

This paper contains 4 sections, 11 equations, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Experimental Evaluation
  4. Conclusions

Figures (3)

  • Figure 1: On the left, the figure depicts the obtained reward during training that is smoothed using a moving average filter with a window of 100. On the right, the action probabilities for each method are depicted using the same moving average setting.
  • Figure 2: Part-based representation of the actor model. At the top and bottom rows, the input and responses of each layer are depicted. In the central row, the weights of the actor model are depicted. The biases are omitted for simplicity.
  • Figure 3: Optimized input of actor model to maximize the action probability of a given action using three different initialization. In the first column, the observations are optimized to maximize the forward action probability. In the second column, the observations are optimized to maximize the backward action probability. A different initialization of the observation vector is used for each row.