Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

R2PS: Worst-Case Robust Real-Time Pursuit Strategies under Partial Observability

Runyu Lu, Ruochuan Shi, Yuanheng Zhu, Dongbin Zhao

TL;DR

This work tackles worst-case robust real-time pursuit under partial observability in graph-based pursuit-evasion games. It develops a DP-based foundation that remains optimal under asynchronous evader moves and extends it with a belief-preservation mechanism to handle partial observability, enabling observation-based pursuer policies. By integrating belief preservation with Equilibrium Policy Generalization (EPG), the authors train a cross-graph RL pursuer capable of zero-shot generalization to unseen graphs, achieving robust performance against adversarial, asynchronous evaders. Empirical results show substantial gains over standard RL and PSRO baselines, especially as graph size grows and observation ranges improve, highlighting practical viability for real-world security scenarios.

Abstract

Computing worst-case robust strategies in pursuit-evasion games (PEGs) is time-consuming, especially when real-world factors like partial observability are considered. While important for general security purposes, real-time applicable pursuit strategies for graph-based PEGs are currently missing when the pursuers only have imperfect information about the evader's position. Although state-of-the-art reinforcement learning (RL) methods like Equilibrium Policy Generalization (EPG) and Grasper provide guidelines for learning graph neural network (GNN) policies robust to different game dynamics, they are restricted to the scenario of perfect information and do not take into account the possible case where the evader can predict the pursuers' actions. This paper introduces the first approach to worst-case robust real-time pursuit strategies (R2PS) under partial observability. We first prove that a traditional dynamic programming (DP) algorithm for solving Markov PEGs maintains optimality under the asynchronous moves by the evader. Then, we propose a belief preservation mechanism about the evader's possible positions, extending the DP pursuit strategies to a partially observable setting. Finally, we embed the belief preservation into the state-of-the-art EPG framework to finish our R2PS learning scheme, which leads to a real-time pursuer policy through cross-graph reinforcement learning against the asynchronous-move DP evasion strategies. After reinforcement learning, our policy achieves robust zero-shot generalization to unseen real-world graph structures and consistently outperforms the policy directly trained on the test graphs by the existing game RL approach.

R2PS: Worst-Case Robust Real-Time Pursuit Strategies under Partial Observability

TL;DR

This work tackles worst-case robust real-time pursuit under partial observability in graph-based pursuit-evasion games. It develops a DP-based foundation that remains optimal under asynchronous evader moves and extends it with a belief-preservation mechanism to handle partial observability, enabling observation-based pursuer policies. By integrating belief preservation with Equilibrium Policy Generalization (EPG), the authors train a cross-graph RL pursuer capable of zero-shot generalization to unseen graphs, achieving robust performance against adversarial, asynchronous evaders. Empirical results show substantial gains over standard RL and PSRO baselines, especially as graph size grows and observation ranges improve, highlighting practical viability for real-world security scenarios.

Abstract

Computing worst-case robust strategies in pursuit-evasion games (PEGs) is time-consuming, especially when real-world factors like partial observability are considered. While important for general security purposes, real-time applicable pursuit strategies for graph-based PEGs are currently missing when the pursuers only have imperfect information about the evader's position. Although state-of-the-art reinforcement learning (RL) methods like Equilibrium Policy Generalization (EPG) and Grasper provide guidelines for learning graph neural network (GNN) policies robust to different game dynamics, they are restricted to the scenario of perfect information and do not take into account the possible case where the evader can predict the pursuers' actions. This paper introduces the first approach to worst-case robust real-time pursuit strategies (R2PS) under partial observability. We first prove that a traditional dynamic programming (DP) algorithm for solving Markov PEGs maintains optimality under the asynchronous moves by the evader. Then, we propose a belief preservation mechanism about the evader's possible positions, extending the DP pursuit strategies to a partially observable setting. Finally, we embed the belief preservation into the state-of-the-art EPG framework to finish our R2PS learning scheme, which leads to a real-time pursuer policy through cross-graph reinforcement learning against the asynchronous-move DP evasion strategies. After reinforcement learning, our policy achieves robust zero-shot generalization to unseen real-world graph structures and consistently outperforms the policy directly trained on the test graphs by the existing game RL approach.

Paper Structure

This paper contains 31 sections, 6 theorems, 28 equations, 6 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Problem Formulation
  4. Dynamic Programming for Markov PEGs
  5. Extending Dynamic Programming Policies to Asynchronous Moves and Partial Observability
  6. Asynchronous-Move Setting
  7. Finding Robust Real-Time Pursuit Strategies (R2PS) via Adversarial Reinforcement Learning across Graphs
  8. Adversarial Reinforcement Learning
  9. Implementation and Complexity Analysis
  10. Evaluations
  11. Evaluations of Extended DP Pursuers
  12. Evaluations of Generalized RL Pursuers
  13. Scalability Tests and Ablation Studies
  14. Conclusion
  15. Omitted Proofs
  16. ...and 16 more sections

Key Result

Theorem 1

If there exists a pure-strategy Nash equilibrium in the Markov PEG, then the joint policy $({{\mu }^{*}},{{\nu }^{*}})$ defined by (2-1) and (2-2) is a Nash equilibrium.

Figures (6)

  • Figure 1: Cross-Graph Reinforcement Learning of Generalized Pursuer Policy
  • Figure 2: Pursuit Initialization under Limited Observation Range (Nodes with Blue Outlines)
  • Figure 3: Pursuit Illustration under Belief Preservation (Shadowed Area around Evader)
  • Figure 4: Cross-Graph Learning Curves of Generalized Pursuer Policies
  • Figure 5: Illustration of Test Graphs (Starting from Scotland-Yard Map)
  • ...and 1 more figures

Theorems & Definitions (12)

  • Theorem 1
  • Lemma 1
  • Theorem 2
  • Corollary 1
  • Theorem 3
  • Lemma 2
  • proof
  • proof
  • proof
  • proof
  • ...and 2 more