Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sample-Efficient Neurosymbolic Deep Reinforcement Learning

Celeste Veronese, Daniele Meli, Alessandro Farinelli

TL;DR

Addresses the sample-inefficiency and generalization gaps of deep reinforcement learning in long-horizon, sparse-reward tasks. The method, SR-DQN, augments standard DRL with background symbolic knowledge encoded as partial policies in Answer Set Programming and uses online reasoning to bias exploration and rescale Q-values $Q(s,a)$. It introduces two mechanisms—SR-Exploration and SR-Exploitation—driven by a confidence parameter $ ho$ and an $oldsymbol{ abla}$-decay (epsilon) schedule to balance symbolic guidance and neural learning, validated on OfficeWorld and DoorKey where it outperforms reward-machine baselines. The results show improved convergence, robustness to imperfect priors, and a scalable framework for integrating symbolic reasoning with DRL, with future work extending to policy-gradient methods and richer logical representations.

Abstract

Reinforcement Learning (RL) is a well-established framework for sequential decision-making in complex environments. However, state-of-the-art Deep RL (DRL) algorithms typically require large training datasets and often struggle to generalize beyond small-scale training scenarios, even within standard benchmarks. We propose a neuro-symbolic DRL approach that integrates background symbolic knowledge to improve sample efficiency and generalization to more challenging, unseen tasks. Partial policies defined for simple domain instances, where high performance is easily attained, are transferred as useful priors to accelerate learning in more complex settings and avoid tuning DRL parameters from scratch. To do so, partial policies are represented as logical rules, and online reasoning is performed to guide the training process through two mechanisms: (i) biasing the action distribution during exploration, and (ii) rescaling Q-values during exploitation. This neuro-symbolic integration enhances interpretability and trustworthiness while accelerating convergence, particularly in sparse-reward environments and tasks with long planning horizons. We empirically validate our methodology on challenging variants of gridworld environments, both in the fully observable and partially observable setting. We show improved performance over a state-of-the-art reward machine baseline.

Sample-Efficient Neurosymbolic Deep Reinforcement Learning

TL;DR

Addresses the sample-inefficiency and generalization gaps of deep reinforcement learning in long-horizon, sparse-reward tasks. The method, SR-DQN, augments standard DRL with background symbolic knowledge encoded as partial policies in Answer Set Programming and uses online reasoning to bias exploration and rescale Q-values . It introduces two mechanisms—SR-Exploration and SR-Exploitation—driven by a confidence parameter and an -decay (epsilon) schedule to balance symbolic guidance and neural learning, validated on OfficeWorld and DoorKey where it outperforms reward-machine baselines. The results show improved convergence, robustness to imperfect priors, and a scalable framework for integrating symbolic reasoning with DRL, with future work extending to policy-gradient methods and richer logical representations.

Abstract

Reinforcement Learning (RL) is a well-established framework for sequential decision-making in complex environments. However, state-of-the-art Deep RL (DRL) algorithms typically require large training datasets and often struggle to generalize beyond small-scale training scenarios, even within standard benchmarks. We propose a neuro-symbolic DRL approach that integrates background symbolic knowledge to improve sample efficiency and generalization to more challenging, unseen tasks. Partial policies defined for simple domain instances, where high performance is easily attained, are transferred as useful priors to accelerate learning in more complex settings and avoid tuning DRL parameters from scratch. To do so, partial policies are represented as logical rules, and online reasoning is performed to guide the training process through two mechanisms: (i) biasing the action distribution during exploration, and (ii) rescaling Q-values during exploitation. This neuro-symbolic integration enhances interpretability and trustworthiness while accelerating convergence, particularly in sparse-reward environments and tasks with long planning horizons. We empirically validate our methodology on challenging variants of gridworld environments, both in the fully observable and partially observable setting. We show improved performance over a state-of-the-art reward machine baseline.
Paper Structure (23 sections, 8 equations, 4 figures, 1 table, 3 algorithms)

This paper contains 23 sections, 8 equations, 4 figures, 1 table, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Background
  4. MDPs and Reinforcement Learning
  5. Domains
  6. Answer Set Programming and Reasoning
  7. Methodology
  8. Logical Representation of the MDP
  9. Logical Representation of Policy Knowledge
  10. Neuro-symbolic training
  11. Neuro-symbolic exploration
  12. Neuro-symbolic exploitation
  13. Impact of symbolic knowledge on training convergence
  14. Complexity of symbolic inference in training
  15. Empirical Evaluation
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Our testing domains: OfficeWorld (left) and DoorKey (right).
  • Figure 2: OfficeWorld results on the DeliverCoffeeAndMail task (left) and on the PatrolABC task (right).
  • Figure 3: Training results on the DoorKey environment in random maps, varying grid size and number of keys.
  • Figure 4: Ablation study over the different components of the SR-DQN algorithm, namely SR-Exploration and SR-Exploitation, compared to the baselines and the full SR-DQN algorithm (left) and training curve of SR-DQN algorithm with either different $\epsilon_f$ and $\epsilon_r$ (center), or different $\rho$ values (right). All studies are performed on $8\times8$ DoorKey maps with 4 keys.