Policy Mirror Descent with Lookahead

TL;DR

This work proposes a new class of PMD algorithms called h-PMD which incorporates multi-step greedy policy improvement with lookahead depth $h$ to the PMD update rule, and establishes a sample complexity for $h-PMD which improves over prior work.

Abstract

Policy Mirror Descent (PMD) stands as a versatile algorithmic framework encompassing several seminal policy gradient algorithms such as natural policy gradient, with connections with state-of-the-art reinforcement learning (RL) algorithms such as TRPO and PPO. PMD can be seen as a soft Policy Iteration algorithm implementing regularized 1-step greedy policy improvement. However, 1-step greedy policies might not be the best choice and recent remarkable empirical successes in RL such as AlphaGo and AlphaZero have demonstrated that greedy approaches with respect to multiple steps outperform their 1-step counterpart. In this work, we propose a new class of PMD algorithms called $h$-PMD which incorporates multi-step greedy policy improvement with lookahead depth $h$ to the PMD update rule. To solve discounted infinite horizon Markov Decision Processes with discount factor $γ$, we show that $h$-PMD which generalizes the standard PMD enjoys a faster dimension-free $γ^h$-linear convergence rate, contingent on the computation of multi-step greedy policies. We propose an inexact version of $h$-PMD where lookahead action values are estimated. Under a generative model, we establish a sample complexity for $h$-PMD which improves over prior work. Finally, we extend our result to linear function approximation to scale to large state spaces. Under suitable assumptions, our sample complexity only involves dependence on the dimension of the feature map space instead of the state space size.

Figures (12)

• Figure 1: Suboptimality value function gap for $h$-PMD in the exact (left) and inexact (middle/right) settings, plotted against iterations in the exact case (left) and against both iterations (middle) and runtime (right) in the inexact case. 16 runs performed for each $h$, mean in solid line and standard deviation as shaded area. In dotted lines (left), the step size $\eta_k$ is equal to its lower bound in sec. \ref{['sec:exact-h-pmd']}, with the choice $c_k := \gamma^{2h (k + 1)}$ (note the dependence on $h$) and in solid lines, the step size $\eta_k$ is set using an identical stepsize schedule across all values of $h$ with $c_k := \gamma^{2 (k + 1)}$ to isolate the effect of the lookahead. Notice that higher values of $h$ still perform better even in terms of runtime.
• Figure 2: Suboptimality value function gap for $h$-PMD using Euclidean Bregman divergence. in the exact (left) and inexact (right) settings. (Right) We performed 32 runs for each value of $h$, the mean is shown as a solid line and the standard deviation as a shaded area.
• Figure 3: Samples used by inexact $h$-PMD at each iteration. When compared with the first figure, it is clear to see the algorithm needs less samples to converge with higher values of $h$. Since far less iterations are needed with higher values of lookahead, the benefit of higher convergence rate greatly outweighs the additional cost per iteration.
• Figure 4: Value function gap against running time. Note that the algorithms with higher values for $h$ still converge faster in terms of runtime.
• Figure 5: Behaviour of our adaptive stepsize $\eta_k$ at each iteration of the algorithm for different values of $h$. Note that the stepsize does not diverge towards infinity, instead it usually vanishes after a certain number of iterations.

Theorems & Definitions (37)

• Remark 3.1
• Theorem 4.1: Exact $h$-PMD
• Remark 5.1
• Remark 5.2
• Theorem 5.3: Inexact $h$-PMD
• Theorem 5.4: Sample complexity of $h$-PMD
• Theorem 6.3: Convergence of $h$-PMD with linear function approximation
• Remark 7.1
• Lemma A.1
• proof