Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Convergence of Policy in Unregularized Policy Mirror Descent

Dachao Lin, Zhihua Zhang

TL;DR

The paper analyzes policy convergence under unregularized Policy Mirror Descent using generalized Bregman divergences, establishing explicit sublinear and linear rates for the value function and linking them to policy convergence. It proves that, under certain conditions, PMD can exhibit faster, even finite-step, convergence to an optimal policy, with Euclidean distance enabling explicit finite-step stopping and other divergences offering analogous accelerated behavior under suitable step-size schemes. The results connect policy convergence to value-function convergence and extend prior asymptotic findings to concrete bounds, while illustrating how common divergences (e.g., KL, Euclidean, Tsallis) influence update dynamics and convergence. The empirical section corroborates the theoretical claims and highlights practical trade-offs between convergence speed and per-step computation across divergences and learning-rate schedules.

Abstract

In this short note, we give the convergence analysis of the policy in the recent famous policy mirror descent (PMD). We mainly consider the unregularized setting following [11] with generalized Bregman divergence. The difference is that we directly give the convergence rates of policy under generalized Bregman divergence. Our results are inspired by the convergence of value function in previous works and are an extension study of policy mirror descent. Though some results have already appeared in previous work, we further discover a large body of Bregman divergences could give finite-step convergence to an optimal policy, such as the classical Euclidean distance.

On the Convergence of Policy in Unregularized Policy Mirror Descent

TL;DR

The paper analyzes policy convergence under unregularized Policy Mirror Descent using generalized Bregman divergences, establishing explicit sublinear and linear rates for the value function and linking them to policy convergence. It proves that, under certain conditions, PMD can exhibit faster, even finite-step, convergence to an optimal policy, with Euclidean distance enabling explicit finite-step stopping and other divergences offering analogous accelerated behavior under suitable step-size schemes. The results connect policy convergence to value-function convergence and extend prior asymptotic findings to concrete bounds, while illustrating how common divergences (e.g., KL, Euclidean, Tsallis) influence update dynamics and convergence. The empirical section corroborates the theoretical claims and highlights practical trade-offs between convergence speed and per-step computation across divergences and learning-rate schedules.

Abstract

In this short note, we give the convergence analysis of the policy in the recent famous policy mirror descent (PMD). We mainly consider the unregularized setting following [11] with generalized Bregman divergence. The difference is that we directly give the convergence rates of policy under generalized Bregman divergence. Our results are inspired by the convergence of value function in previous works and are an extension study of policy mirror descent. Though some results have already appeared in previous work, we further discover a large body of Bregman divergences could give finite-step convergence to an optimal policy, such as the classical Euclidean distance.
Paper Structure (9 sections, 6 theorems, 41 equations, 1 figure, 1 table)

This paper contains 9 sections, 6 theorems, 41 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Notation of MDP
  4. Policy Mirror Descent
  5. Sublinear/Linear Convergence of Value Function
  6. Faster Convergence of Policy
  7. An Example of Euclidean Distance
  8. Application to Common Divergence
  9. Experiments

Key Result

Theorem 3.1

Consider the generalized policy mirror descent method with $\pi(0) \in \mathrm{ri}(\Pi)$ and constant step size $\eta_k = \eta>0, \forall k \geq 0$. For any $\rho \in \Delta({\mathcal{S}})$, we have for all $k \geq 0$,

Figures (1)

  • Figure 1: L2: the squared Euclidean distance; KL: Kullback-Leibler (KL) divergence. Default: exponential learning rates. 'reg': using adaptive regularization $\tau_k$ with $1 + \eta_k\tau_k = 1/\gamma$. VI: value iteration. 'clr': constant learning rate.

Theorems & Definitions (11)

  • Theorem 3.1: xiao2022convergence Theorem 8
  • Theorem 3.2: xiao2022convergence Theorem 10
  • Proposition 1
  • Remark 3.3
  • Proposition 2
  • Definition 4.1: Inherited from khodadadian2021linear Definitions 3.1 and 3.2
  • Remark 4.2
  • Theorem 4.3
  • Remark 4.4
  • Theorem 4.5
  • ...and 1 more