Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning from Less: Guiding Deep Reinforcement Learning with Differentiable Symbolic Planning

Zihan Ye, Oleg Arenz, Kristian Kersting

TL;DR

The paper tackles the data-efficiency bottleneck of reinforcement learning in sparse-reward settings by introducing Dylan, a differentiable symbolic planner that can serve as both a reward model and a high-level planner. Dylan encodes planning rules as differentiable tensors, uses a learnable weight matrix with softmax to compose rules, and applies forward-chaining inference to generate plan valuations that shape rewards or guide hierarchical actions. The key contributions are (1) a differentiable symbolic planner integrated into RL as an interpretable reward model, (2) adaptive reward shaping that leverages the full distribution of candidate plans, and (3) a differentiable planner capable of adaptivity and compositional generalization to solve unseen tasks without retraining. Empirical results in MiniGrid show faster convergence and better generalization for PPO and A2C when aided by Dylan, especially under harder, sparser-reward conditions, highlighting Dylan’s potential to bridge symbolic reasoning and reinforcement learning in a scalable, interpretable way.

Abstract

When tackling complex problems, humans naturally break them down into smaller, manageable subtasks and adjust their initial plans based on observations. For instance, if you want to make coffee at a friend's place, you might initially plan to grab coffee beans, go to the coffee machine, and pour them into the machine. Upon noticing that the machine is full, you would skip the initial steps and proceed directly to brewing. In stark contrast, state of the art reinforcement learners, such as Proximal Policy Optimization (PPO), lack such prior knowledge and therefore require significantly more training steps to exhibit comparable adaptive behavior. Thus, a central research question arises: \textit{How can we enable reinforcement learning (RL) agents to have similar ``human priors'', allowing the agent to learn with fewer training interactions?} To address this challenge, we propose differentiable symbolic planner (Dylan), a novel framework that integrates symbolic planning into Reinforcement Learning. Dylan serves as a reward model that dynamically shapes rewards by leveraging human priors, guiding agents through intermediate subtasks, thus enabling more efficient exploration. Beyond reward shaping, Dylan can work as a high level planner that composes primitive policies to generate new behaviors while avoiding common symbolic planner pitfalls such as infinite execution loops. Our experimental evaluations demonstrate that Dylan significantly improves RL agents' performance and facilitates generalization to unseen tasks.

Learning from Less: Guiding Deep Reinforcement Learning with Differentiable Symbolic Planning

TL;DR

The paper tackles the data-efficiency bottleneck of reinforcement learning in sparse-reward settings by introducing Dylan, a differentiable symbolic planner that can serve as both a reward model and a high-level planner. Dylan encodes planning rules as differentiable tensors, uses a learnable weight matrix with softmax to compose rules, and applies forward-chaining inference to generate plan valuations that shape rewards or guide hierarchical actions. The key contributions are (1) a differentiable symbolic planner integrated into RL as an interpretable reward model, (2) adaptive reward shaping that leverages the full distribution of candidate plans, and (3) a differentiable planner capable of adaptivity and compositional generalization to solve unseen tasks without retraining. Empirical results in MiniGrid show faster convergence and better generalization for PPO and A2C when aided by Dylan, especially under harder, sparser-reward conditions, highlighting Dylan’s potential to bridge symbolic reasoning and reinforcement learning in a scalable, interpretable way.

Abstract

When tackling complex problems, humans naturally break them down into smaller, manageable subtasks and adjust their initial plans based on observations. For instance, if you want to make coffee at a friend's place, you might initially plan to grab coffee beans, go to the coffee machine, and pour them into the machine. Upon noticing that the machine is full, you would skip the initial steps and proceed directly to brewing. In stark contrast, state of the art reinforcement learners, such as Proximal Policy Optimization (PPO), lack such prior knowledge and therefore require significantly more training steps to exhibit comparable adaptive behavior. Thus, a central research question arises: \textit{How can we enable reinforcement learning (RL) agents to have similar ``human priors'', allowing the agent to learn with fewer training interactions?} To address this challenge, we propose differentiable symbolic planner (Dylan), a novel framework that integrates symbolic planning into Reinforcement Learning. Dylan serves as a reward model that dynamically shapes rewards by leveraging human priors, guiding agents through intermediate subtasks, thus enabling more efficient exploration. Beyond reward shaping, Dylan can work as a high level planner that composes primitive policies to generate new behaviors while avoiding common symbolic planner pitfalls such as infinite execution loops. Our experimental evaluations demonstrate that Dylan significantly improves RL agents' performance and facilitates generalization to unseen tasks.
Paper Structure (27 sections, 22 equations, 5 figures, 2 tables)

This paper contains 27 sections, 22 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Dylan: Guiding RL using Differentiable Symbolic Planning
  4. Dylan as Reward Model
  5. Dylan as Adaptive Reward Model
  6. Dylan as Differentiable Planner
  7. Experimental Evaluation
  8. Dylan as reward model
  9. Dylan as adaptive reward model
  10. Dylan as differentiable planner
  11. Dylan's adaptivity.
  12. Dylan's compositionality.
  13. Limitations
  14. Conclusions
  15. Acknowledgements
  16. ...and 12 more sections

Figures (5)

  • Figure 1: An overview of Dylan working as a reward model. Given the goal and game rules, Dylan first generates candidate plans to achieve the goal and then shapes rewards based on the generated plans, balancing among all candidate plans or selecting the best one to follow through.
  • Figure 2: Dylan, as a differentiable symbolic planner, is capable of adapting to tasks that require different search strategies and stitching together different primitive policies to generate novel behaviours. (Best viewed in color)
  • Figure 3: Dylan boost RL agents’ performances as a static reward model. Return comparisons of methods with and without Dylan in $\mathtt{12\times12}$ and $\mathtt{16\times16}$ MiniGrid environments during training. The return curves are averaged over three runs, with the solid lines representing the mean values and the shaded areas indicating the minimum and maximum values. All curves are smoothed using exponential moving average (EMA) for improved readability. (Best viewed in color)
  • Figure 4: Dylan further boost RL agents' performances as an adaptive reward model, shown by the returns over training of the PPO baseline, and PPO with reasoner and adaptive reasoner, on the $\mathtt{8\times8}$ MiniGrid-DoorKey environment using static rewards, adaptive rewards, and pure PPO. Averages over $3$ runs; solid lines show means, shaded areas show standard error. (Best viewed in color)
  • Figure 5: Training loss curve as Dylan learns to adapt to multiple tasks requiring different search strategies. Results are averaged over three runs, with solid lines indicating mean values and shaded areas representing standard deviation.