Three Pathways to Neurosymbolic Reinforcement Learning with Interpretable Model and Policy Networks
Peter Graf, Patrick Emami
TL;DR
This work addresses how to build reinforcement learning agents that are both differentiable and interpretable by integrating neurosymbolic architectures into NSRL. It presents three pathways—model-free RL with Differentiable Decision Trees, model-based RL using Logical Neural Networks to form planning problems, and differentiable predictive control with LNNs in a differentiable simulator—demonstrated in building energy management with the OCHRE/Gym environment. The study highlights key tradeoffs: while differentiability aids learning, classical logic is discrete and can hinder optimization; interpretability tends to favor symbolic rules, yet scalability and discretization pose challenges. The results show that rule-based controllers remain strong baselines in some TOU scenarios, but learned NSAI policies can adapt and offer interpretability benefits, especially when warm-starts and symbolic planning are employed, pointing to a promising fusion of symbolic and continuous approaches for real-world control. The work lays groundwork for scalable NSRL in complex, uncertain systems and suggests directions for future enhancements, including large language model integration to guide rule discovery and refinement.
Abstract
Neurosymbolic AI combines the interpretability, parsimony, and explicit reasoning of classical symbolic approaches with the statistical learning of data-driven neural approaches. Models and policies that are simultaneously differentiable and interpretable may be key enablers of this marriage. This paper demonstrates three pathways to implementing such models and policies in a real-world reinforcement learning setting. Specifically, we study a broad class of neural networks that build interpretable semantics directly into their architecture. We reveal and highlight both the potential and the essential difficulties of combining logic, simulation, and learning. One lesson is that learning benefits from continuity and differentiability, but classical logic is discrete and non-differentiable. The relaxation to real-valued, differentiable representations presents a trade-off; the more learnable, the less interpretable. Another lesson is that using logic in the context of a numerical simulation involves a non-trivial mapping from raw (e.g., real-valued time series) simulation data to logical predicates. Some open questions this note exposes include: What are the limits of rule-based controllers, and how learnable are they? Do the differentiable interpretable approaches discussed here scale to large, complex, uncertain systems? Can we truly achieve interpretability? We highlight these and other themes across the three approaches.