Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Markov Decision Processes with Terminal Law Constraints

Bianca Marin Moreno, Margaux Brégère, Pierre Gaillard, Nadia Oudjane

TL;DR

This work introduces a reset-free, periodic framework for online MDPs with terminal-law constraints, where policies must return the system to its initial state distribution after a fixed period. It formalizes periodic policies and periodic regret, and develops two algorithms, MDPP-K and MDPP-U, to compute approximately periodic policies under known and unknown initial distributions, respectively. The authors prove non-asymptotic, sublinear periodic regret of order $\tilde{O}(T^{3/4})$ in both settings for $M>1$ homogeneous agents, with detailed analyses of the MD terms, feasibility, and error propagation from unknown dynamics. Empirical results on grid-world tasks validate the approach and demonstrate practical benefits over episodic baselines, highlighting the framework’s potential for real-world reset-free control problems such as energy-demand management and autonomous robotics.

Abstract

Traditional reinforcement learning usually assumes either episodic interactions with resets or continuous operation to minimize average or cumulative loss. While episodic settings have many theoretical results, resets are often unrealistic in practice. The infinite-horizon setting avoids this issue but lacks non-asymptotic guarantees in online scenarios with unknown dynamics. In this work, we move towards closing this gap by introducing a reset-free framework called the periodic framework, where the goal is to find periodic policies: policies that not only minimize cumulative loss but also return the agents to their initial state distribution after a fixed number of steps. We formalize the problem of finding optimal periodic policies and identify sufficient conditions under which it is well-defined for tabular Markov decision processes. To evaluate algorithms in this framework, we introduce the periodic regret, a measure that balances cumulative loss with the terminal law constraint. We then propose the first algorithms for computing periodic policies in two multi-agent settings and show they achieve sublinear periodic regret of order $\tilde O(T^{3/4})$. This provides the first non-asymptotic guarantees for reset-free learning in the setting of $M$ homogeneous agents, for $M > 1$.

Online Markov Decision Processes with Terminal Law Constraints

TL;DR

This work introduces a reset-free, periodic framework for online MDPs with terminal-law constraints, where policies must return the system to its initial state distribution after a fixed period. It formalizes periodic policies and periodic regret, and develops two algorithms, MDPP-K and MDPP-U, to compute approximately periodic policies under known and unknown initial distributions, respectively. The authors prove non-asymptotic, sublinear periodic regret of order in both settings for homogeneous agents, with detailed analyses of the MD terms, feasibility, and error propagation from unknown dynamics. Empirical results on grid-world tasks validate the approach and demonstrate practical benefits over episodic baselines, highlighting the framework’s potential for real-world reset-free control problems such as energy-demand management and autonomous robotics.

Abstract

Traditional reinforcement learning usually assumes either episodic interactions with resets or continuous operation to minimize average or cumulative loss. While episodic settings have many theoretical results, resets are often unrealistic in practice. The infinite-horizon setting avoids this issue but lacks non-asymptotic guarantees in online scenarios with unknown dynamics. In this work, we move towards closing this gap by introducing a reset-free framework called the periodic framework, where the goal is to find periodic policies: policies that not only minimize cumulative loss but also return the agents to their initial state distribution after a fixed number of steps. We formalize the problem of finding optimal periodic policies and identify sufficient conditions under which it is well-defined for tabular Markov decision processes. To evaluate algorithms in this framework, we introduce the periodic regret, a measure that balances cumulative loss with the terminal law constraint. We then propose the first algorithms for computing periodic policies in two multi-agent settings and show they achieve sublinear periodic regret of order . This provides the first non-asymptotic guarantees for reset-free learning in the setting of homogeneous agents, for .
Paper Structure (49 sections, 23 theorems, 133 equations, 4 figures, 5 tables, 4 algorithms)

This paper contains 49 sections, 23 theorems, 133 equations, 4 figures, 5 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. The learning problem
  3. Preliminaries: finite-horizon (convex) MDP
  4. Periodic policies
  5. Learning frameworks
  6. Framework 1: known initial distributions
  7. Preliminaries
  8. Algorithm
  9. Periodic regret analysis
  10. Framework 2: unknown initial distributions
  11. Experiments
  12. List of Notations
  13. Related Work
  14. Autonomous RL:
  15. Stochastic control with constraints in law:
  16. ...and 34 more sections

Key Result

Lemma 2

For any $\bar{\alpha} \geq \alpha$, for $\alpha$ as in Ass. ass:contraction, the problem in Eq. iteration_md_solver is feasible with high-probability.

Figures (4)

  • Figure 1: [left] Initial agent dist.; [right] Obstacles (reward in yellow, constraints in blue).
  • Figure 2: Periodic regret for each environment.
  • Figure 3: Max-entropy: state dist. after $5000$ eps. of MDPP-K [up] and Bonus MD-CURL [down].
  • Figure 4: Obstacles: state dist. after $1000$ eps. for MDPP-K [up] and Bonus MD-CURL [down].

Theorems & Definitions (24)

  • Definition 1: Periodic policy
  • Lemma 2
  • Lemma 3
  • Theorem 4
  • Proposition 5
  • Proposition 6
  • Proposition 7
  • Theorem 8
  • Lemma 9: Lem. 17 of UCRL-2
  • Lemma 10
  • ...and 14 more