Interpretable end-to-end Neurosymbolic Reinforcement Learning agents

TL;DR

Deep RL agents often rely on opaque representations, hindering interpretability. This paper instantiates SCoBots, a neurosymbolic framework that decomposes policy into interpretable object extraction, relation extraction, and action selection, with a neural policy distilled into a symbolic rule-set via ECLAIRE. The implementation leverages SPACE+MOC for unsupervised object representation, a relation extractor for object-relational concepts, and PPO for learning, followed by rule distillation to achieve interpretability. Experiments on Atari games (via OCAtari) show that the approach can yield interpretable policies with competitive performance under certain configurations, while also highlighting the trade-offs between object-extractor accuracy and downstream task performance. Overall, the work advances end-to-end interpretable RL by integrating object-centric learning with symbolic policy distillation, offering a path toward trustworthy, inspectable agents in real-world settings.

Abstract

Deep reinforcement learning (RL) agents rely on shortcut learning, preventing them from generalizing to slightly different environments. To address this problem, symbolic method, that use object-centric states, have been developed. However, comparing these methods to deep agents is not fair, as these last operate from raw pixel-based states. In this work, we instantiate the symbolic SCoBots framework. SCoBots decompose RL tasks into intermediate, interpretable representations, culminating in action decisions based on a comprehensible set of object-centric relational concepts. This architecture aids in demystifying agent decisions. By explicitly learning to extract object-centric representations from raw states, object-centric RL, and policy distillation via rule extraction, this work places itself within the neurosymbolic AI paradigm, blending the strengths of neural networks with symbolic AI. We present the first implementation of an end-to-end trained SCoBot, separately evaluate of its components, on different Atari games. The results demonstrate the framework's potential to create interpretable and performing RL systems, and pave the way for future research directions in obtaining end-to-end interpretable RL agents.

Figures (6)

• Figure 1: Overview of the SCoBots framework.SCoBots decompose the policy into three consecutive steps (i.e., object extraction, relation extraction, action selection) using intermediate Interpretable Concept Bottlenecks (ICB). This enables external users to inspect how the SCoBot agent selects its action. Figure adapted from Delfosse2024InterpretableCB.
• Figure 2: Overview of MOC applied to SPACE. SPACE learns to extract objects using a VAE architecture. The reconstruction problem is split into a foreground and a background component. The foreground latent space is composed of positional and feature information about the objects in the image. The MOC training scheme adds the motion supervision loss and the object continuity loss to improve a base detection model, SPACE in this case. The motion supervision loss is designed to enhance the localization capabilities by guiding the locations (loc) with motion data of the input image. The object continuity loss is applied to feature encodings (enc) of the objects in consecutive images with the goal of ensuring consistency of the encodings representing the same entity.
• Figure 3: Overview of the implementation. The input images are fed consecutively into the object extractor. The resulting objects represented by location and encoding are classified into a specific object class of the respective game. Also, object identity between consecutive frames is inferred via object tracking. This results in a set of objects with properties, including time-related ones. The relation extractor computes inter- or intra-object relations. Finally, the action selector decides, which action to take using an interpretable algorithm.
• Figure 4: Visualization of the games used in our experimental evaluation. The Boxing, Skiing, and Pong Atari environments are shown from left to right. While Boxing and Pong were also used for RL evaluation, the difficult credit assignment game Skiing is only used for object detection.
• Figure 5: Object extractors for Boxing and Pong demonstrate high performance, while Skiing shows lower performance. Precision, recall, and F-score for the combined object extractors of SPACE+MOC and classifier are shown. Results are averages with standard deviations across five SPACE+MOC models trained with different seeds and their corresponding classifiers.