Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Approximate Information States for Worst-Case Control and Learning in Uncertain Systems

Aditya Dave, Nishanth Venkatesh, Andreas A. Malikopoulos

TL;DR

This work addresses robust, worst-case decision-making under partial observations in non-stochastic uncertain systems by introducing information states and their approximate counterparts. The authors develop a dynamic-programming framework that uses an information-state $oldsymbol{}_t$ to achieve optimal planning, and extend it to approximate information states $oldsymbol{} Pi_t$ that can be learned from outputs with provable bounded loss via an approximate DP. They provide explicit conditions and Lipschitz-based bounds that control approximation error, and illustrate the approach through state-quantization examples and learning-based methods in wall-defense and pursuit-evasion problems. The results offer a principled path to tractable worst-case control and robust reinforcement learning under partial observations, with demonstrated computational and performance benefits in representative scenarios.

Abstract

In this paper, we investigate discrete-time decision-making problems in uncertain systems with partially observed states. We consider a non-stochastic model, where uncontrolled disturbances acting on the system take values in bounded sets with unknown distributions. We present a general framework for decision-making in such problems by using the notion of the information state and approximate information state, and introduce conditions to identify an uncertain variable that can be used to compute an optimal strategy through a dynamic program (DP). Next, we relax these conditions and define approximate information states that can be learned from output data without knowledge of system dynamics. We use approximate information states to formulate a DP that yields a strategy with a bounded performance loss. Finally, we illustrate the application of our results in control and reinforcement learning using numerical examples.

Approximate Information States for Worst-Case Control and Learning in Uncertain Systems

TL;DR

This work addresses robust, worst-case decision-making under partial observations in non-stochastic uncertain systems by introducing information states and their approximate counterparts. The authors develop a dynamic-programming framework that uses an information-state to achieve optimal planning, and extend it to approximate information states that can be learned from outputs with provable bounded loss via an approximate DP. They provide explicit conditions and Lipschitz-based bounds that control approximation error, and illustrate the approach through state-quantization examples and learning-based methods in wall-defense and pursuit-evasion problems. The results offer a principled path to tractable worst-case control and robust reinforcement learning under partial observations, with demonstrated computational and performance benefits in representative scenarios.

Abstract

In this paper, we investigate discrete-time decision-making problems in uncertain systems with partially observed states. We consider a non-stochastic model, where uncontrolled disturbances acting on the system take values in bounded sets with unknown distributions. We present a general framework for decision-making in such problems by using the notion of the information state and approximate information state, and introduce conditions to identify an uncertain variable that can be used to compute an optimal strategy through a dynamic program (DP). Next, we relax these conditions and define approximate information states that can be learned from output data without knowledge of system dynamics. We use approximate information states to formulate a DP that yields a strategy with a bounded performance loss. Finally, we illustrate the application of our results in control and reinforcement learning using numerical examples.
Paper Structure (19 sections, 16 theorems, 59 equations, 5 figures)

This paper contains 19 sections, 16 theorems, 59 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Contributions and Organization
  4. Modeling Framework
  5. Preliminaries
  6. Problem Formulation
  7. Dynamic Programs and Information States
  8. Information States
  9. Alternate Characterization of Information States
  10. Examples of Information States
  11. Approximate Information States
  12. Properties of Approximate Information States
  13. Alternate Characterization
  14. Examples
  15. Learning an approximate information state
  16. ...and 4 more sections

Key Result

Theorem 1

Let $\Pi_t = \sigma_t(M_t)$ be an information state at any $t$. Then, for all $t$, and for all $m_t \in [[M_t]]$ and $u_t \in [[U_t]]$,

Figures (5)

  • Figure 1: The wall defense problem with the initial conditions $x_0^{\text{ag}} = (0,2)$ and $y_0 = (0,-2)$.
  • Figure 2: Costs and run-times for $5\times10^3$ simulations and $T=6$.
  • Figure 3: The pursuit evasion problem with the initial conditions $x_0^{\text{ag}} = (0,2)$ and $y_0 = (3, -4)$.
  • Figure 4: The neural network architecture for approximate information states at any $t=0,\dots,T-1$.
  • Figure 5: Worst-case costs for $10^3$ simulations

Theorems & Definitions (47)

  • Remark 1
  • Remark 2
  • Remark 3
  • Remark 4
  • Remark 5
  • Definition 1
  • Theorem 1
  • proof
  • Remark 6
  • Lemma 1
  • ...and 37 more