Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Local Linearity: the Key for No-regret Reinforcement Learning in Continuous MDPs

Davide Maran, Alberto Maria Metelli, Matteo Papini, Marcello Restelli

TL;DR

Cinderella, a no-regret algorithm for this general representation class of Markov Decision Processes, is introduced and Cinderella is shown to achieve state-of-the-art regret bounds for all previously known (and some new) continuous MDPs for which RL is learnable and feasible.

Abstract

Achieving the no-regret property for Reinforcement Learning (RL) problems in continuous state and action-space environments is one of the major open problems in the field. Existing solutions either work under very specific assumptions or achieve bounds that are vacuous in some regimes. Furthermore, many structural assumptions are known to suffer from a provably unavoidable exponential dependence on the time horizon $H$ in the regret, which makes any possible solution unfeasible in practice. In this paper, we identify local linearity as the feature that makes Markov Decision Processes (MDPs) both learnable (sublinear regret) and feasible (regret that is polynomial in $H$). We define a novel MDP representation class, namely Locally Linearizable MDPs, generalizing other representation classes like Linear MDPs and MDPS with low inherent Belmman error. Then, i) we introduce Cinderella, a no-regret algorithm for this general representation class, and ii) we show that all known learnable and feasible MDP families are representable in this class. We first show that all known feasible MDPs belong to a family that we call Mildly Smooth MDPs. Then, we show how any mildly smooth MDP can be represented as a Locally Linearizable MDP by an appropriate choice of representation. This way, Cinderella is shown to achieve state-of-the-art regret bounds for all previously known (and some new) continuous MDPs for which RL is learnable and feasible.

Local Linearity: the Key for No-regret Reinforcement Learning in Continuous MDPs

TL;DR

Cinderella, a no-regret algorithm for this general representation class of Markov Decision Processes, is introduced and Cinderella is shown to achieve state-of-the-art regret bounds for all previously known (and some new) continuous MDPs for which RL is learnable and feasible.

Abstract

Achieving the no-regret property for Reinforcement Learning (RL) problems in continuous state and action-space environments is one of the major open problems in the field. Existing solutions either work under very specific assumptions or achieve bounds that are vacuous in some regimes. Furthermore, many structural assumptions are known to suffer from a provably unavoidable exponential dependence on the time horizon in the regret, which makes any possible solution unfeasible in practice. In this paper, we identify local linearity as the feature that makes Markov Decision Processes (MDPs) both learnable (sublinear regret) and feasible (regret that is polynomial in ). We define a novel MDP representation class, namely Locally Linearizable MDPs, generalizing other representation classes like Linear MDPs and MDPS with low inherent Belmman error. Then, i) we introduce Cinderella, a no-regret algorithm for this general representation class, and ii) we show that all known learnable and feasible MDP families are representable in this class. We first show that all known feasible MDPs belong to a family that we call Mildly Smooth MDPs. Then, we show how any mildly smooth MDP can be represented as a Locally Linearizable MDP by an appropriate choice of representation. This way, Cinderella is shown to achieve state-of-the-art regret bounds for all previously known (and some new) continuous MDPs for which RL is learnable and feasible.

Paper Structure

This paper contains 31 sections, 22 theorems, 157 equations, 2 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and set-up
  3. Locally Linearizable MDPs
  4. Algorithm
  5. From local linearity to Mildly Smooth MDPs
  6. Comparison with related works
  7. Conclusion
  8. Table of Notation
  9. Locally Linearizable MDPs
  10. Definition of the quasi-optimal solution
  11. Algorithm
  12. Algorithm analysis
  13. Additional notation
  14. Decomposition of the estimated solution
  15. Failure event
  16. ...and 16 more sections

Key Result

Theorem 2

Assume to be in an $\mathcal{I}-$Locally Linearizable MDP with $L_\phi=\mathcal{O}(1)$, $\sup_{n\in[N_h]}\mathcal{R}_{h,n}=\mathcal{O}(\sqrt d_h)$ and that Assumption ass:normalized holds. Then, with probability at least $1-\delta$, Cinderella (Algorithm alg:Cinderella), with $\lambda=1$ achieves a

Figures (2)

  • Figure 1: In Locally Linearizable MDPs, we have that, as shown in (a), the space $\mathcal{Z}$ is partitioned into several regions, which do not need to be convex nor connected. On each of these regions, as shown in (b), the result of the Bellman optimality operator can be well approximated by a $Q$ function that is linear in the feature map, with a parameter $\theta$ that may depend on the region itself.
  • Figure 2: Relation between the setting described in this paper and the other settings proposed for reinforcement learning in continuous state-action spaces. The dashed line means that inclusion holds, but passing to the larger family brings a $\text{exp}(H)$ lower bound on the regret. As we can see, the Mildly smooth MDP is the largest known setting for which regret of order $\text{poly(H)}$ is possible. Note that the Strongly Smooth family also contains known families like LQRs and Linear MDPs with smooth feature map maran2024no.

Theorems & Definitions (42)

  • Definition 1
  • Definition 2
  • Theorem 2
  • Definition 3
  • Theorem 3
  • Theorem 4
  • Corollary 5
  • proof
  • Definition 4
  • Theorem 6
  • ...and 32 more