Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online MDP with Transition Prototypes: A Robust Adaptive Approach

Shuo Sun, Meng Qi, Zuo-Jun Max Shen

TL;DR

The paper proposes an online robust MDP framework leveraging a finite set of transition prototypes to adaptively identify the true kernel while guaranteeing robust policy performance. The RPO-AAS algorithm updates an adaptive ambiguity set across prototypes and computes the robust policy via backward induction, achieving sublinear regret and finite-sample guarantees, with an explicit stopping criterion for prototype convergence. Theoretical results bound regret and establish convergence to the true model, and numerical experiments in a GridWorld demonstrate improved early-stage performance and robustness compared to non-robust and standard online RL baselines. This work advances model-based RL under structured prior information by balancing exploration, exploitation, and robustness in an online setting with provable guarantees.

Abstract

In this work, we consider an online robust Markov Decision Process (MDP) where we have the information of finitely many prototypes of the underlying transition kernel. We consider an adaptively updated ambiguity set of the prototypes and propose an algorithm that efficiently identifies the true underlying transition kernel while guaranteeing the performance of the corresponding robust policy. To be more specific, we provide a sublinear regret of the subsequent optimal robust policy. We also provide an early stopping mechanism and a worst-case performance bound of the value function. In numerical experiments, we demonstrate that our method outperforms existing approaches, particularly in the early stage with limited data. This work contributes to robust MDPs by considering possible prior information about the underlying transition probability and online learning, offering both theoretical insights and practical algorithms for improved decision-making under uncertainty.

Online MDP with Transition Prototypes: A Robust Adaptive Approach

TL;DR

The paper proposes an online robust MDP framework leveraging a finite set of transition prototypes to adaptively identify the true kernel while guaranteeing robust policy performance. The RPO-AAS algorithm updates an adaptive ambiguity set across prototypes and computes the robust policy via backward induction, achieving sublinear regret and finite-sample guarantees, with an explicit stopping criterion for prototype convergence. Theoretical results bound regret and establish convergence to the true model, and numerical experiments in a GridWorld demonstrate improved early-stage performance and robustness compared to non-robust and standard online RL baselines. This work advances model-based RL under structured prior information by balancing exploration, exploitation, and robustness in an online setting with provable guarantees.

Abstract

In this work, we consider an online robust Markov Decision Process (MDP) where we have the information of finitely many prototypes of the underlying transition kernel. We consider an adaptively updated ambiguity set of the prototypes and propose an algorithm that efficiently identifies the true underlying transition kernel while guaranteeing the performance of the corresponding robust policy. To be more specific, we provide a sublinear regret of the subsequent optimal robust policy. We also provide an early stopping mechanism and a worst-case performance bound of the value function. In numerical experiments, we demonstrate that our method outperforms existing approaches, particularly in the early stage with limited data. This work contributes to robust MDPs by considering possible prior information about the underlying transition probability and online learning, offering both theoretical insights and practical algorithms for improved decision-making under uncertainty.

Paper Structure

This paper contains 36 sections, 12 theorems, 40 equations, 4 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation and Preliminaries
  4. Problem Formulation
  5. Loop-Free MDP.
  6. Transition Prototypes.
  7. Preliminaries
  8. Occupancy measures.
  9. The RPO-AAS Algorithm
  10. Theoretical Results
  11. Analysis of Regret
  12. Finite-Sample Guarantee and Convergence
  13. Extend to Non-robust Algorithm: Selecting the Best Candidate
  14. Numerical Experiments
  15. Prototype configuration.
  16. ...and 21 more sections

Key Result

Lemma 1

For the ambiguity set updated as described in Algorithm alg:update-U, the true transition kernel lies in the ambiguity set $\mathcal{U}_t$, i.e., $P_0\in \mathcal{U}_t$ for all $t\in [T]$ with probability at least $1-\delta$.

Figures (4)

  • Figure 1: Average Expected Episode Rewards of different algorithms with Fixed-gap Prototypes when $K=4$.
  • Figure 2: Average Expected Episode Rewards of different algorithms with Fixed-gap Prototypes when $K=10$.
  • Figure 3: Average Expected Episode Rewards of different algorithms with 4 Random Prototypes.
  • Figure 4: Average Expected Episode Rewards of different algorithms with 10 Random Prototypes.

Theorems & Definitions (20)

  • Lemma 1
  • Proposition 1
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • Theorem 1
  • Theorem 2: Finite-sample guarantee
  • Theorem 3: Prototype Ambiguity Set Convergence
  • Lemma 6
  • ...and 10 more