Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Symbolic Persistent Macro-Actions for POMDP Solving Over Time

Celeste Veronese, Daniele Meli, Alessandro Farinelli

TL;DR

This work tackles scalable decision-making under uncertainty by learning persistent macro-actions through Inductive Logic Programming and Event Calculus to bias online POMDP planners. The EC-based theories ground macro-actions from execution traces, enabling time-aware abstraction to guide Monte Carlo Tree Search in POMCP and DESPOT. The approach yields improved expressiveness and generalization, along with substantial computational efficiency gains on Rocksample and Pocman benchmarks. The results demonstrate that symbolic, interpretable temporal heuristics can outperform purely time-independent methods and approach handcrafted policies, with potential for extension to richer logics and real-world robotics tasks.

Abstract

This paper proposes an integration of temporal logical reasoning and Partially Observable Markov Decision Processes (POMDPs) to achieve interpretable decision-making under uncertainty with macro-actions. Our method leverages a fragment of Linear Temporal Logic (LTL) based on Event Calculus (EC) to generate \emph{persistent} (i.e., constant) macro-actions, which guide Monte Carlo Tree Search (MCTS)-based POMDP solvers over a time horizon, significantly reducing inference time while ensuring robust performance. Such macro-actions are learnt via Inductive Logic Programming (ILP) from a few traces of execution (belief-action pairs), thus eliminating the need for manually designed heuristics and requiring only the specification of the POMDP transition model. In the Pocman and Rocksample benchmark scenarios, our learned macro-actions demonstrate increased expressiveness and generality when compared to time-independent heuristics, indeed offering substantial computational efficiency improvements.

Learning Symbolic Persistent Macro-Actions for POMDP Solving Over Time

TL;DR

This work tackles scalable decision-making under uncertainty by learning persistent macro-actions through Inductive Logic Programming and Event Calculus to bias online POMDP planners. The EC-based theories ground macro-actions from execution traces, enabling time-aware abstraction to guide Monte Carlo Tree Search in POMCP and DESPOT. The approach yields improved expressiveness and generalization, along with substantial computational efficiency gains on Rocksample and Pocman benchmarks. The results demonstrate that symbolic, interpretable temporal heuristics can outperform purely time-independent methods and approach handcrafted policies, with potential for extension to richer logics and real-world robotics tasks.

Abstract

This paper proposes an integration of temporal logical reasoning and Partially Observable Markov Decision Processes (POMDPs) to achieve interpretable decision-making under uncertainty with macro-actions. Our method leverages a fragment of Linear Temporal Logic (LTL) based on Event Calculus (EC) to generate \emph{persistent} (i.e., constant) macro-actions, which guide Monte Carlo Tree Search (MCTS)-based POMDP solvers over a time horizon, significantly reducing inference time while ensuring robust performance. Such macro-actions are learnt via Inductive Logic Programming (ILP) from a few traces of execution (belief-action pairs), thus eliminating the need for manually designed heuristics and requiring only the specification of the POMDP transition model. In the Pocman and Rocksample benchmark scenarios, our learned macro-actions demonstrate increased expressiveness and generality when compared to time-independent heuristics, indeed offering substantial computational efficiency improvements.
Paper Structure (26 sections, 8 equations, 3 figures, 1 table)

This paper contains 26 sections, 8 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Partially Observable Markov Decision Processes (POMDPs)
  5. Online POMDP planning
  6. Answer Set Programming and Event Calculus
  7. Inductive Logic Programming
  8. Case studies
  9. Rocksample
  10. Pocman
  11. Methodology
  12. ASP formalization of the POMDP problem
  13. Macro-actions generation from execution traces
  14. Integration of macro-actions as planning heuristics
  15. Experimental results
  16. ...and 11 more sections

Figures (3)

  • Figure 1: Planning performances (mean $\pm$ std) in rocksample, in POMCP varying $N$ (left) or the number of simulations (center), and DESPOT (right) varying $N$.
  • Figure 2: POMCP performances in the pocman scenario with $17\times 19$ grid size and $G=4$, both in nominal (left) and challenging (right) conditions.
  • Figure 3: Comparison with AIS on rocksample with $M=4$ (left) and $M=8$ (right). Second row shows training results on the same settings.