Analysis of Off-Policy Multi-Step TD-Learning with Linear Function Approximation
Donghwan Lee
TL;DR
The paper tackles instability in off-policy TD learning with linear function approximation by analyzing $n$-step methods. It first develops model-based deterministic analogues (n-step projected value iteration, gradient-based formulations, and a system-operator framework) and proves contraction or strong convexity when the horizon $n$ is large enough, yielding a well-defined fixed point $ heta_*^n$. Building on these results, it introduces two model-free stochastic algorithms, $n$-TD and $n$-GTD, and shows they converge to $ heta_*^n$ for sufficiently large $n$ under standard step-size assumptions. Collectively, the work provides convergence guarantees for $n$-step TD methods under the deadly triad, bridging deterministic analyses with stochastic RL and guiding practical selection of $n$ to ensure stable learning.
Abstract
This paper analyzes multi-step TD-learning algorithms within the `deadly triad' scenario, characterized by linear function approximation, off-policy learning, and bootstrapping. In particular, we prove that n-step TD-learning algorithms converge to a solution as the sampling horizon n increases sufficiently. The paper is divided into two parts. In the first part, we comprehensively examine the fundamental properties of their model-based deterministic counterparts, including projected value iteration, gradient descent algorithms, and the control theoretic approach, which can be viewed as prototype deterministic algorithms whose analysis plays a pivotal role in understanding and developing their model-free reinforcement learning counterparts. In particular, we prove that these algorithms converge to meaningful solutions when n is sufficiently large. Based on these findings, two n-step TD-learning algorithms are proposed and analyzed, which can be seen as the model-free reinforcement learning counterparts of the gradient and control theoretic algorithms.