End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanations

TL;DR

This work tackles explainability and sample efficiency in neuro-symbolic reinforcement learning by proposing INSIGHT, which jointly learns manageable structured states and coordinate-based symbolic policies from visual inputs. It achieves this by distilling vision foundation models into a scalable perception module, coupling it with an Extrapolation-based Logic (EQL) policy learner guided by a neural actor, and augmenting it with a GPT-4 powered textual explanations pipeline grounded through concept grounding. The approach yields state refinement with reward signals, competitive Atari results, and meaningful natural-language explanations for policies and decisions. The findings suggest that end-to-end refinement of structured states, combined with neural guidance and accessible explanations, enhances both performance and transparency in NS-RL, with practical implications for trustworthy AI.

Abstract

Neuro-symbolic reinforcement learning (NS-RL) has emerged as a promising paradigm for explainable decision-making, characterized by the interpretability of symbolic policies. NS-RL entails structured state representations for tasks with visual observations, but previous methods cannot refine the structured states with rewards due to a lack of efficiency. Accessibility also remains an issue, as extensive domain knowledge is required to interpret symbolic policies. In this paper, we present a neuro-symbolic framework for jointly learning structured states and symbolic policies, whose key idea is to distill the vision foundation model into an efficient perception module and refine it during policy learning. Moreover, we design a pipeline to prompt GPT-4 to generate textual explanations for the learned policies and decisions, significantly reducing users' cognitive load to understand the symbolic policies. We verify the efficacy of our approach on nine Atari tasks and present GPT-generated explanations for policies and decisions.

Figures (9)

• Figure 1: For tasks with visual input, INSIGHT can simultaneously learn the coordinates of objects in observations and coordinate-based symbolic policies simultaneously, and it can interpret learned policies and specific decisions in natural language. $\text{y}_\text{agent,i}$ represents the vertical coordinate of the agent in the $i^{th}$ frame. Both policy interpretation and decision explanation are produced by entering the policy and a predefined prompt template into the LLM.
• Figure 2: INSIGHT consists of three components: a perception module, a policy learning module, and a policy explanation module. The perception module learns to predict object coordinates using a frame-symbol dataset distilled from vision foundation models. The policy learning module is responsible for learning coordinate-based symbolic policies. In particular, to address with the limited expressiveness of object coordinates, it uses a neural actor to interact with the environment. The policy explanation module can generate policy interpretations and decision explanations using task description, policy description, and values of object coordinates.
• Figure 3: Each component of INSIGHT is critical for overall performance. Detailed performance analyses of INSIGHT and its variants across five tasks are presented. Refer to \ref{['remain_methodablation']} for results for the remaining four tasks. Test returns are normalized so that INSIGHT corresponds to one and random policy is zero. Fixing the perception module during policy learning (i.e., Fixed) hinders performance for all tasks, indicating that it is crucial to refine states with reward signals. The results of w/o Pretrain and w/o NG show that pre-training the perception module and the proposed neural guidance scheme also improves performance.
• Figure 4: INSIGHT demonstrates robustness to hyper-parameters. Examining the influence of sparsity regularization weight $\lambda_\text{reg}$, the EQL actor's width/layers, and the weight of $\mathcal{L}_\text{cnn}$ on SpaceInvaders performance. Results for additional tasks are available in \ref{['hyper ablation remain']}. Overall, INSIGHT shows substantial robustness to variations in hyper-parameters.
• Figure 5: Examples for textual explanations for Pong. Readers may refer to \ref{['Comprehensive Prompt Template Overview']} for full prompts. Left: interpretations for a learned policy. The interpretations identify influential input variables and summarize triggering patterns of actions. Right: explanations for an action taken at a state. The four images located at the bottom illustrate the state. The motion of the ball and the opponent's paddle are deduced from input variables, which are used for supporting explanations of actions.