The Role of Target Update Frequencies in Q-Learning

TL;DR

This work analyzes target network maintenance in Q-learning through an approximate dynamic programming lens, treating periodic target updates as a nested Bellman-operator approximation with an inner SGD loop. It proves that fixed TUF schedules are provably suboptimal and derives a near-optimal increasing TUF strategy with geometry $K_n \propto \gamma^{-2n/3}$ that improves the sample complexity from $O\left(\frac{\log((1-\gamma)^{-1}\varepsilon^{-1})}{(1-\gamma)^4\xi^2\varepsilon^2}\right)$ to $O\left(\frac{1}{\xi^2(1-\gamma)^5\varepsilon^2}\right)$, effectively removing the logarithmic dependence on $\varepsilon$. The analysis combines a general outer-loop contraction with a detailed inner-loop SGD bound, under asynchronous sampling, yielding explicit bias-variance trade-offs for the target-update period. The results inform principled design of target-update schedules in tabular Q-learning and offer a foundation for extensions to deeper RL settings, including accuracy-triggered updates and adaptive optimizers. Overall, the paper provides both finite-time convergence guarantees and practical guidance for stabilizing and accelerating Q-learning through adaptive target updates.

Abstract

The target network update frequency (TUF) is a central stabilization mechanism in (deep) Q-learning. However, their selection remains poorly understood and is often treated merely as another tunable hyperparameter rather than as a principled design decision. This work provides a theoretical analysis of target fixing in tabular Q-learning through the lens of approximate dynamic programming. We formulate periodic target updates as a nested optimization scheme in which each outer iteration applies an inexact Bellman optimality operator, approximated by a generic inner loop optimizer. Rigorous theory yields a finite-time convergence analysis for the asynchronous sampling setting, specializing to stochastic gradient descent in the inner loop. Our results deliver an explicit characterization of the bias-variance trade-off induced by the target update period, showing how to optimally set this critical hyperparameter. We prove that constant target update schedules are suboptimal, incurring a logarithmic overhead in sample complexity that is entirely avoidable with adaptive schedules. Our analysis shows that the optimal target update frequency increases geometrically over the course of the learning process.

Figures (6)

• Figure 1: Top row: Lunar Lander (Algo: DQN with different constant target update frequency (TUF) using SGD with custom learning rate schedule that linearly decreases from $0.01$ to $0.0001$ over each target-freezing interval) and a GridWorld from Appendix \ref{['app:GW']} (Algo: Q-learning with target freezing and our theory guided learning rate $1/(1+k/(2|\mathcal{S}||\mathcal{A}|))$ with $|\mathcal{S}||\mathcal{A}|=52$ over each target-freezing interval). Bottom row: Same environments, same algorithms but increasing TUF (ITUF), initialised in fixed TUF from top row.
• Figure 2: Q-learning on GridWorld from Appendix \ref{['app:GW']} with optimally increasing target update frequencies (ITUF) from Theorem \ref{['thm:pql-growing-cl']} in comparison to Q-learning with fixed target update frequencies (TUF) for $K\in\{1000,10000,100000\}$. Shaded bands indicate uncertainty $95\%$ confidence intervals across 50 random seeds: Small fixed TUFs lead to early plateaus due to coarse Bellman operator approximations, while large fixed TUFs converge slowly because overly accurate inner-loop approximations dominate early iterations. Geometrically increasing schedule predicted by our theory avoids this trade-off by balancing contraction and Bellman approximation error, achieving both rapid initial progress and asymptotic convergence.
• Figure 3: Grid World environment used for numerical experiments. The agent starts at state S (gray) and aims to reach the goal state G (green) while avoiding the bomb states B (red). The orange states represent the high variance region. The optimal path is indicated by the gray arrows.
• Figure 4: Q-learning on GridWorld from Appendix \ref{['app:GW']} with optimally increasing target update frequencies (ITUF) from Theorem \ref{['thm:pql-growing-cl']} in comparison to Q-learning with fixed target update frequencies (TUF) for $K\in\{1000,10000,100000\}$. The discount factor in the top row is set to $\gamma=0.9$, below $\gamma=0.95$. Shaded bands in the first column indicate uncertainty $95\%$ confidence intervals across 50 random seeds: Small fixed TUFs lead to early plateaus due to coarse Bellman operator approximations, while large fixed TUFs converge slowly because overly accurate inner-loop approximations dominate early iterations. Geometrically increasing schedule predicted by our theory avoids this trade-off by balancing contraction and Bellman approximation error, achieving both rapid initial progress and asymptotic convergence, leading to the choice of the optimal path during evaluation.
• Figure 5: Q-learning with accuracy-triggered target updates, $K_{\min} \in\{100,1000,10000\}$ and $K_{\max}=1e6$ on the GridWorld environment from Appendix \ref{['app:GW']}. Target accuracies are choosen by $\varepsilon_n = \tfrac{1}{n^2}$, motivated by the condition for convergence in Prop. \ref{['thm:approx-contraction']}. As comparison we include Q-learning with $\text{TUF}=K_{\max}=1e6$ to illustrate the effect of accuracy-triggered target updates. Unnecessarily long inner loops for Bellman approximations are stopped early leading to an improved convergence behavior.

Theorems & Definitions (20)

• Theorem 1.1: Informal
• Remark 1.2
• Proposition 4.1: Approximate contraction of the outer loop
• proof
• Corollary 4.2
• Remark 4.4
• Theorem 4.5: Inner loop convergence rate
• Corollary 4.6
• Theorem 5.1: Fixed TUF
• Remark 5.2