Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Performance Guarantees for Data-Driven Sequential Decision-Making

Bowen Li, Edwin K. P. Chong, Ali Pezeshki

Abstract

The solutions to many sequential decision-making problems are characterized by dynamic programming and Bellman's principle of optimality. However, due to the inherent complexity of solving Bellman's equation exactly, there has been significant interest in developing various approximate dynamic programming (ADP) schemes to obtain near-optimal solutions. A fundamental question that arises is: how close are the objective values produced by ADP schemes relative to the true optimal objective values? In this paper, we develop a general framework that provides performance guarantees for ADP schemes in the form of ratio bounds. Specifically, we show that the objective value under an ADP scheme is at least a computable fraction of the optimal value. We further demonstrate the applicability of our theoretical framework through two applications: data-driven robot path planning and multi-agent sensor coverage.

Performance Guarantees for Data-Driven Sequential Decision-Making

Abstract

The solutions to many sequential decision-making problems are characterized by dynamic programming and Bellman's principle of optimality. However, due to the inherent complexity of solving Bellman's equation exactly, there has been significant interest in developing various approximate dynamic programming (ADP) schemes to obtain near-optimal solutions. A fundamental question that arises is: how close are the objective values produced by ADP schemes relative to the true optimal objective values? In this paper, we develop a general framework that provides performance guarantees for ADP schemes in the form of ratio bounds. Specifically, we show that the objective value under an ADP scheme is at least a computable fraction of the optimal value. We further demonstrate the applicability of our theoretical framework through two applications: data-driven robot path planning and multi-agent sensor coverage.
Paper Structure (19 sections, 1 theorem, 52 equations, 6 figures, 1 table)

This paper contains 19 sections, 1 theorem, 52 equations, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Stochastic Optimal Control
  4. Dynamic Programming with Bellman's Equation
  5. Approximate Dynamic Programming
  6. Previous Results
  7. Main Results
  8. Performance Guarantee
  9. Connection to Submodular Optimization
  10. Existing Results
  11. Generalization
  12. Applications
  13. Robot Path Planning via Linear Quadratic Gaussian (LQG)
  14. Problem Setup
  15. Experiments
  16. ...and 4 more sections

Key Result

Theorem 1

Assuming we have and $\beta = \hat{V} / \overline{V} \leq \hat{V} / V^{*}.$

Figures (6)

  • Figure 1: Robot path planning
  • Figure 2: Comparison of training labels in stepwise error functions.
  • Figure 3: Upper figure: Comparison of $V^{*}$,$\hat{V}$, and $\underline{V}$; Lower figure: Comparison of the ratio bounds.
  • Figure 4: Sensor coverage for event detection in a mission space.
  • Figure 5: Mission space partition.
  • ...and 1 more figures

Theorems & Definitions (14)

  • Definition 1
  • Remark 1
  • Definition 2
  • Remark 2
  • Definition 3
  • Remark 3
  • Definition 4
  • Remark 4
  • Definition 5
  • Remark 5
  • ...and 4 more