Explainable Reinforcement Learning via a Causal World Model
Zhongwei Yu, Jingqing Ruan, Dengpeng Xing
TL;DR
This work tackles explainable reinforcement learning by learning a causal world model that requires no prior causal structure. It combines causal discovery to identify a sparse SCM with attention-based inference networks to extract action influences, enabling causal-chain explanations that trace how actions affect states and rewards. The approach yields an AIM-like explanation mechanism while remaining accurate enough to support model-based RL, demonstrated on continuous and discrete domains and showing competitive performance relative to dense baselines. By linking explainability with policy learning through a shared causal model, the method offers faithful explanations and practical improvements for safe, interpretable RL in unknown environments.
Abstract
Generating explanations for reinforcement learning (RL) is challenging as actions may produce long-term effects on the future. In this paper, we develop a novel framework for explainable RL by learning a causal world model without prior knowledge of the causal structure of the environment. The model captures the influence of actions, allowing us to interpret the long-term effects of actions through causal chains, which present how actions influence environmental variables and finally lead to rewards. Different from most explanatory models which suffer from low accuracy, our model remains accurate while improving explainability, making it applicable in model-based learning. As a result, we demonstrate that our causal model can serve as the bridge between explainability and learning.