Towards Interpretable Deep Reinforcement Learning Models via Inverse Reinforcement Learning
Sean Xie, Soroush Vosoughi, Saeed Hassanpour
TL;DR
The paper tackles the challenge of interpreting deep reinforcement learning models by introducing a global, AIRL-based framework that learns a discriminator capturing the reward structure underlying expert behavior. It trains a DRL expert and an AIRL-based novice of identical architecture, then analyzes the discriminator's rewards across trajectories to reveal global decision-making patterns, including how word characteristics relate to rewards in abstractive summarization. The authors apply the method to the CNN/Daily Mail abstractive summarization task, report ROUGE-based performance, and present analyses showing that certain parts of speech and word properties are more rewarded, yielding interpretable, trajectory-level explanations. This approach demonstrates the potential of IRL to provide meaningful, global explanations for complex NLP models and suggests broad applicability to other generative tasks while acknowledging limitations such as data requirements and dependence on the expert's quality. The framework offers a practical path toward increased trust and transparency in DRL systems by translating model behavior into human-understandable reward-based patterns.
Abstract
Artificial intelligence, particularly through recent advancements in deep learning, has achieved exceptional performances in many tasks in fields such as natural language processing and computer vision. In addition to desirable evaluation metrics, a high level of interpretability is often required for these models to be reliably utilized. Therefore, explanations that offer insight into the process by which a model maps its inputs onto its outputs are much sought-after. Unfortunately, the current black box nature of machine learning models is still an unresolved issue and this very nature prevents researchers from learning and providing explicative descriptions for a model's behavior and final predictions. In this work, we propose a novel framework utilizing Adversarial Inverse Reinforcement Learning that can provide global explanations for decisions made by a Reinforcement Learning model and capture intuitive tendencies that the model follows by summarizing the model's decision-making process.