Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Policy Mirror Descent with Temporal Difference Learning: Sample Complexity under Online Markov Data

Wenye Li, Hongxu Chen, Jiacai Liu, Ke Wei

TL;DR

This work analyzes policy mirror descent (PMD) in reinforcement learning under online Markov data with temporal-difference (TD) learning. It introduces two TD-based PMD variants—Expected TD-PMD (off-policy) and Approximate TD-PMD (mixed-policy)—and proves competitive sample complexities: a standard $ ilde{O}( rac{1}{ ho^2})$ for average-time $ ext{ε}$-optimality and an $O( rac{1}{ε^2})$ last-iterate rate using adaptive policy-update steps, all in the tabular setting. A key novelty is bounding stochastic bias without inner-loop critic refinement by employing adaptive batch sizes and a weighted Bellman operator, yielding an $O( rac{1}{ε^2})$ sample complexity free of polylog factors. The results extend known PMD/TD-PMD analyses to practical Markov data scenarios, closely matching generative-model performance and offering a batch-Q-learning equivalence under special mirror maps. Together, the findings advance understanding of sample efficiency for PMD-type methods in online RL with TD criticism and provide actionable guidance for adaptive-step strategies.

Abstract

This paper studies the policy mirror descent (PMD) method, which is a general policy optimization framework in reinforcement learning and can cover a wide range of policy gradient methods by specifying difference mirror maps. Existing sample complexity analysis for policy mirror descent either focuses on the generative sampling model, or the Markovian sampling model but with the action values being explicitly approximated to certain pre-specified accuracy. In contrast, we consider the sample complexity of policy mirror descent with temporal difference (TD) learning under the Markovian sampling model. Two algorithms called Expected TD-PMD and Approximate TD-PMD have been presented, which are off-policy and mixed policy algorithms respectively. Under a small enough constant policy update step size, the $\tilde{O}(\varepsilon^{-2})$ (a logarithm factor about $\varepsilon$ is hidden in $\tilde{O}(\cdot)$) sample complexity can be established for them to achieve average-time $\varepsilon$-optimality. The sample complexity is further improved to $O(\varepsilon^{-2})$ (without the hidden logarithm factor) to achieve the last-iterate $\varepsilon$-optimality based on adaptive policy update step sizes.

Policy Mirror Descent with Temporal Difference Learning: Sample Complexity under Online Markov Data

TL;DR

This work analyzes policy mirror descent (PMD) in reinforcement learning under online Markov data with temporal-difference (TD) learning. It introduces two TD-based PMD variants—Expected TD-PMD (off-policy) and Approximate TD-PMD (mixed-policy)—and proves competitive sample complexities: a standard for average-time -optimality and an last-iterate rate using adaptive policy-update steps, all in the tabular setting. A key novelty is bounding stochastic bias without inner-loop critic refinement by employing adaptive batch sizes and a weighted Bellman operator, yielding an sample complexity free of polylog factors. The results extend known PMD/TD-PMD analyses to practical Markov data scenarios, closely matching generative-model performance and offering a batch-Q-learning equivalence under special mirror maps. Together, the findings advance understanding of sample efficiency for PMD-type methods in online RL with TD criticism and provide actionable guidance for adaptive-step strategies.

Abstract

This paper studies the policy mirror descent (PMD) method, which is a general policy optimization framework in reinforcement learning and can cover a wide range of policy gradient methods by specifying difference mirror maps. Existing sample complexity analysis for policy mirror descent either focuses on the generative sampling model, or the Markovian sampling model but with the action values being explicitly approximated to certain pre-specified accuracy. In contrast, we consider the sample complexity of policy mirror descent with temporal difference (TD) learning under the Markovian sampling model. Two algorithms called Expected TD-PMD and Approximate TD-PMD have been presented, which are off-policy and mixed policy algorithms respectively. Under a small enough constant policy update step size, the (a logarithm factor about is hidden in ) sample complexity can be established for them to achieve average-time -optimality. The sample complexity is further improved to (without the hidden logarithm factor) to achieve the last-iterate -optimality based on adaptive policy update step sizes.
Paper Structure (27 sections, 34 theorems, 175 equations, 2 figures, 1 table, 3 algorithms)

This paper contains 27 sections, 34 theorems, 175 equations, 2 figures, 1 table, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related works
  3. PMD
  4. Natural policy gradient
  5. Other related policy gradient methods
  6. Motivation and main contributions
  7. Preliminary
  8. More on MDP
  9. More on TD-PMD
  10. Expected TD-PMD with Off-Policy Markov Data
  11. Expected TD-PMD with constant step size
  12. Expected TD-PMD with adaptive step sizes
  13. Batch Q-learning
  14. Approximate TD-PMD with Mixed-Policy Markov Data
  15. Approximate TD-PMD with constant step size
  16. ...and 12 more sections

Key Result

Lemma 2.1

For any $\pi\in\Pi$, $V\in\mathbb{R}^{|\mathcal{S}|}$, $Q\in\mathbb{R}^{|\mathcal{S}||\mathcal{A}|}$, $\mu\in\Delta(\mathcal{S})$, and $\rho\in\Delta(\mathcal{S}\times\mathcal{A})$, there holds

Figures (2)

  • Figure 1: An illustration of Expected TD-PMD with off-policy Markov data (Algorithm \ref{['alg:Expected-TD-AC-PMD']}),
  • Figure 2: An illustration of Approximate TD-PMD with mixed-policy Markov data (Algorithm \ref{['alg:Approximated-TD-AC-PMD']}).

Theorems & Definitions (55)

  • Lemma 2.1: Performance difference lemma kakade2002approximatelyliu2025tdpmd
  • Lemma 2.2
  • proof
  • Lemma 2.3
  • Proposition 3.3
  • Lemma 3.5
  • proof
  • Lemma 3.6
  • Lemma 3.7
  • Proposition 3.8
  • ...and 45 more