Rank-One Modified Value Iteration

TL;DR

This paper tackles the efficiency of planning and learning in Markov decision processes by introducing Rank-One Value Iteration (R1-VI) and Rank-One Q-Learning (R1-QL). Both methods replace the full policy-evaluation step with a rank-one approximation of the transition matrix, using the stationary distribution approximated via the power method; this yields a PI-type update that preserves the same convergence rate as standard VI/QL while maintaining the same per-iteration computational cost as the first-order methods. Theoretical results establish linear convergence to the optimal value or Q-function, and extensive simulations on Garnet and Graph MDPs show that the proposed methods outperform first-order and accelerated counterparts in practice. The work demonstrates that a carefully designed rank-one correction can significantly speed up planning and learning without increasing per-iteration complexity, offering practical benefits for large-scale MDPs and motivating extensions to broader settings and function-approximation regimes.

Abstract

In this paper, we provide a novel algorithm for solving planning and learning problems of Markov decision processes. The proposed algorithm follows a policy iteration-type update by using a rank-one approximation of the transition probability matrix in the policy evaluation step. This rank-one approximation is closely related to the stationary distribution of the corresponding transition probability matrix, which is approximated using the power method. We provide theoretical guarantees for the convergence of the proposed algorithm to optimal (action-)value function with the same rate and computational complexity as the value iteration algorithm in the planning problem and as the Q-learning algorithm in the learning problem. Through our extensive numerical simulations, however, we show that the proposed algorithm consistently outperforms first-order algorithms and their accelerated versions for both planning and learning problems.

Figures (9)

• Figure 1: Planning algorithms -- the median number of iterations required for each algorithm to reach a fixed error threshold across four discount factors $\gamma$ for the two MDPs.
• Figure 2: Learning algorithms -- the median error values achieved by each learning algorithm over the course of 5000 iterations across four discount factors $\gamma$ for the two MDPs.
• Figure 3: Comparison of the planning algorithms in Garnet MDP with various $\gamma$ values.
• Figure 4: Comparison of the planning algorithms in Graph MDP with various $\gamma$ values.
• Figure 5: Comparison of the learning algorithms in Garnet MDP with various $\gamma$ values.

Theorems & Definitions (18)

• Lemma 3.1: Policy iteration
• Lemma 3.2: Rank-1 approximation
• Theorem 3.3: Convergence of R1-VI
• Remark 3.4: Generalization to modified policy iteration
• Lemma 4.1: Policy iteration for Q-function
• Theorem 4.2: Convergence of R1-QL
• Lemma A.1
• proof
• Lemma A.2
• proof