SymDQN: Symbolic Knowledge and Reasoning in Neural Network-based Reinforcement Learning
Ivo Amador, Nina Gierasimczuk
TL;DR
SymDQN tackles interpretability and control in reinforcement learning by integrating Logic Tensor Networks into a DuelDQN framework. It adds modular symbolic components—ShapeRecognizer, RewardPredictor, ActionReasoner, and ActionFilter—to ground environment structure, predict immediate rewards, and align action selection with symbolic reasoning. An ablation study on a ShapesGridEnv shows that ActionFilter boosts early learning and final performance, while ActionReasoner can improve precision but may limit overall learning, highlighting trade-offs between fast adaptation and strict symbolic alignment. Overall, the work demonstrates a viable neuro-symbolic RL approach and outlines avenues for richer symbolic grounding and broader evaluation.
Abstract
We propose a learning architecture that allows symbolic control and guidance in reinforcement learning with deep neural networks. We introduce SymDQN, a novel modular approach that augments the existing Dueling Deep Q-Networks (DuelDQN) architecture with modules based on the neuro-symbolic framework of Logic Tensor Networks (LTNs). The modules guide action policy learning and allow reinforcement learning agents to display behaviour consistent with reasoning about the environment. Our experiment is an ablation study performed on the modules. It is conducted in a reinforcement learning environment of a 5x5 grid navigated by an agent that encounters various shapes, each associated with a given reward. The underlying DuelDQN attempts to learn the optimal behaviour of the agent in this environment, while the modules facilitate shape recognition and reward prediction. We show that our architecture significantly improves learning, both in terms of performance and the precision of the agent. The modularity of SymDQN allows reflecting on the intricacies and complexities of combining neural and symbolic approaches in reinforcement learning.