Compositional Concept-Based Neuron-Level Interpretability for Deep Reinforcement Learning

TL;DR

This paper tackles the interpretability gap in deep reinforcement learning by introducing a neuron-level, concept-based framework. It formalizes atomic concepts as binary state-space functions and builds compositional concepts via logical operators, then maps these to neuron activations using binarization and Jaccard similarity, with beam search to discover interpretable formulas. The approach is validated on discrete and continuous control tasks (Blackjack-v1, LunarLander-v3, LunarLander-Continuous-v2), revealing interpretable, human-aligned concepts across neurons in the policy/value networks, and is further substantiated by targeted perturbations that trigger predictable activation and action changes. The results demonstrate robust, transferable insights into how neurons encode strategic and control-related concepts, with potential implications for pruning, robustness, and controllable RL systems in real-world applications.

Abstract

Deep reinforcement learning (DRL), through learning policies or values represented by neural networks, has successfully addressed many complex control problems. However, the neural networks introduced by DRL lack interpretability and transparency. Current DRL interpretability methods largely treat neural networks as black boxes, with few approaches delving into the internal mechanisms of policy/value networks. This limitation undermines trust in both the neural network models that represent policies and the explanations derived from them. In this work, we propose a novel concept-based interpretability method that provides fine-grained explanations of DRL models at the neuron level. Our method formalizes atomic concepts as binary functions over the state space and constructs complex concepts through logical operations. By analyzing the correspondence between neuron activations and concept functions, we establish interpretable explanations for individual neurons in policy/value networks. Experimental results on both continuous control tasks and discrete decision-making environments demonstrate that our method can effectively identify meaningful concepts that align with human understanding while faithfully reflecting the network's decision-making logic.

Figures (3)

• Figure 1: Our concept-based interpretation framework for DRL: (1) We design atomic concept functions (e.g., "height near ground", "high velocity") and construct concept vectors $c$ by applying these functions to state sequences; (2) Record DRL model neuron activations $a$ from the policy/value network; (3) Compose concept outputs and match with neuron activations through optimization; (4) Each neuron $(i,l)$ is explained by the compositional concept that best matches its activation pattern (e.g., $(c_1 \vee c_2) \wedge c_3$).
• Figure 2: Visualization of three representative neurons in the discrete LunarLander (DQN) value network. Each row shows three different states demonstrating distinct neuron functionalities: landing detection (Neuron 19, top), attitude control (Neuron 41, middle), and horizontal velocity management (Neuron 21, bottom). Red/green colors indicate active/inactive states, with activation values shown. Key state variables are annotated to highlight triggering conditions.
• Figure 3: Perturbation analysis of Neuron 5 in discrete LunarLander. Left: The network architecture showing how Neuron 5 contributes to action selection through weighted connections. Middle: Original state where the neuron is active ($h(s)=4.00$) and the network selects "fire left engine". Right: Perturbed state where modifying the x-coordinate to -0.24 causes the neuron to become inactive ($h(s')=-4.48$) and changes the selected action to "main engine". The consistent relationship between concept satisfaction, neuron activation, and action selection validates our interpretation.