Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Goal Recognition as Reinforcement Learning

Leonardo Rosa Amado, Reuth Mirsky, Felipe Meneguzzi

TL;DR

This paper reframes goal recognition as a model-free reinforcement learning problem, replacing handcrafted domain models and online planning with offline learning of per-goal $Q$-functions or policies. It introduces the GRAQL instantiation, trains a $Q$-function for each candidate goal, and performs online inference by comparing observations to these goal-specific value structures using three distance measures ($$MaxUtil, KL$$-divergence, and Divergence Point). Across Blocks, Hanoi, and SkGrid domains, GRAQL achieves competitive performance with planning-based recognizers and demonstrates superior robustness to partial observability and noisy data, while maintaining computational efficiency through precomputation. The framework paves the way for scalable GR without explicit domain dynamics, with future work on function approximation and image-based, real-world settings.

Abstract

Most approaches for goal recognition rely on specifications of the possible dynamics of the actor in the environment when pursuing a goal. These specifications suffer from two key issues. First, encoding these dynamics requires careful design by a domain expert, which is often not robust to noise at recognition time. Second, existing approaches often need costly real-time computations to reason about the likelihood of each potential goal. In this paper, we develop a framework that combines model-free reinforcement learning and goal recognition to alleviate the need for careful, manual domain design, and the need for costly online executions. This framework consists of two main stages: Offline learning of policies or utility functions for each potential goal, and online inference. We provide a first instance of this framework using tabular Q-learning for the learning stage, as well as three measures that can be used to perform the inference stage. The resulting instantiation achieves state-of-the-art performance against goal recognizers on standard evaluation domains and superior performance in noisy environments.

Goal Recognition as Reinforcement Learning

TL;DR

This paper reframes goal recognition as a model-free reinforcement learning problem, replacing handcrafted domain models and online planning with offline learning of per-goal -functions or policies. It introduces the GRAQL instantiation, trains a -function for each candidate goal, and performs online inference by comparing observations to these goal-specific value structures using three distance measures (-divergence, and Divergence Point). Across Blocks, Hanoi, and SkGrid domains, GRAQL achieves competitive performance with planning-based recognizers and demonstrates superior robustness to partial observability and noisy data, while maintaining computational efficiency through precomputation. The framework paves the way for scalable GR without explicit domain dynamics, with future work on function approximation and image-based, real-world settings.

Abstract

Most approaches for goal recognition rely on specifications of the possible dynamics of the actor in the environment when pursuing a goal. These specifications suffer from two key issues. First, encoding these dynamics requires careful design by a domain expert, which is often not robust to noise at recognition time. Second, existing approaches often need costly real-time computations to reason about the likelihood of each potential goal. In this paper, we develop a framework that combines model-free reinforcement learning and goal recognition to alleviate the need for careful, manual domain design, and the need for costly online executions. This framework consists of two main stages: Offline learning of policies or utility functions for each potential goal, and online inference. We provide a first instance of this framework using tabular Q-learning for the learning stage, as well as three measures that can be used to perform the inference stage. The resulting instantiation achieves state-of-the-art performance against goal recognizers on standard evaluation domains and superior performance in noisy environments.
Paper Structure (13 sections, 8 equations, 4 figures, 2 tables, 2 algorithms)

This paper contains 13 sections, 8 equations, 4 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. The Goal Recognition as Reinforcement Learning Framework
  4. Goal Recognition as Q-Learning
  5. Learning $\{Q_g\}_{g\in \mathcal{G}}$ using Q-Learning
  6. Measures for Tabular $Q$-functions
  7. Experimental Evaluation
  8. Hyperparameters and Algorithmic Choices
  9. Testing under Full Observability
  10. Testing under Partial Observability and Noise
  11. State-Only and Action-Only Observations
  12. Related Work
  13. Discussion and Conclusion

Figures (4)

  • Figure 1: A comparison of existing model-based approaches for goal recognition (left) and our proposed framework (right). The key changes in our approach are presented in red.
  • Figure 2: The proposed framework for GR as RL. Details of the Goal Recognition as Q-Learning (GRAQL) instantiation appear in italics.
  • Figure 3: Comparison of R&G, MaxUtil, KL, DP by their accuracy, precision, recall, and F-score for full observability.
  • Figure 4: Performance comparison of R&G and MaxUtil with $\observation^s$ (state-only), $\observation^a$ (action-only), and $\observation$ (state-action).

Theorems & Definitions (4)

  • Definition 1: MDP
  • Definition 2: Utility-based Domain Theory
  • Definition 3: Policy-based Domain Theory
  • Definition 4: Goal Recognition Problem