Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Target-Aligned Reinforcement Learning

Leonard S. Pleiss, James Harrison, Maximilian Schiffer

Abstract

Many reinforcement learning algorithms rely on target networks - lagged copies of the online network - to stabilize training. While effective, this mechanism introduces a fundamental stability-recency tradeoff: slower target updates improve stability but reduce the recency of learning signals, hindering convergence speed. We propose Target-Aligned Reinforcement Learning (TARL), a framework that emphasizes transitions for which the target and online network estimates are highly aligned. By focusing updates on well-aligned targets, TARL mitigates the adverse effects of stale target estimates while retaining the stabilizing benefits of target networks. We provide a theoretical analysis demonstrating that target alignment correction accelerates convergence, and empirically demonstrate consistent improvements over standard reinforcement learning algorithms across various benchmark environments.

Target-Aligned Reinforcement Learning

Abstract

Many reinforcement learning algorithms rely on target networks - lagged copies of the online network - to stabilize training. While effective, this mechanism introduces a fundamental stability-recency tradeoff: slower target updates improve stability but reduce the recency of learning signals, hindering convergence speed. We propose Target-Aligned Reinforcement Learning (TARL), a framework that emphasizes transitions for which the target and online network estimates are highly aligned. By focusing updates on well-aligned targets, TARL mitigates the adverse effects of stale target estimates while retaining the stabilizing benefits of target networks. We provide a theoretical analysis demonstrating that target alignment correction accelerates convergence, and empirically demonstrate consistent improvements over standard reinforcement learning algorithms across various benchmark environments.

Paper Structure

This paper contains 28 sections, 3 theorems, 39 equations, 6 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Contribution
  4. Problem statement
  5. Target-Aligned Reinforcement Learning
  6. Intuition
  7. Target Alignment Score
  8. Alignment-based oversampling
  9. Theoretical Analyses
  10. Framework
  11. Implications.
  12. Consistency and Update Efficiency
  13. Latent Alignment via Online Approximation
  14. Conclusion.
  15. Numerical results
  16. ...and 13 more sections

Key Result

Lemma 4.2.1

Let updates be productive with probability $\lambda > 0.5$. If $U_t^{(k)}$ and $U_t^{(k+K)}$ are treated as samples from this distribution, the probability of them agreeing on direction (alignment) is strictly greater than random chance: Consequently, the sign correlation $c$ between successive updates is non-negative ($c \ge 0$).

Figures (6)

  • Figure 1: Stylized visualization of alignment scenarios. We distinguish between updates fully supported by the online network (scenario I & II) and updates that are only partially or not at all supported by the online network (scenario III & IV). We posit that update types I and II are safer than update types III and IV.
  • Figure 2: Performance comparison between and target-aligned on four games of the MinAtar benchmark over five seeds. Curves represent median returns, smoothed over 10 points, and shaded areas indicate interquartile ranges.
  • Figure 3: Performance comparison between and target-aligned on six games of the MuJoCo benchmarks over five seeds. Curves represent median returns, smoothed over 10 points, and shaded areas indicate interquartile ranges.
  • Figure 4: Performance comparison between DDQN and target-aligned DDQN on four games of the MinAtar benchmark. Curves represent median returns, smoothed over 10 points,
  • Figure 5: Per-seed performance comparison between and target-aligned on four games of the MinAtar benchmarks. Curves represent returns, smoothed over 10 points.
  • ...and 1 more figures

Theorems & Definitions (7)

  • Lemma 4.2.1: Nonnegative Alignment
  • Lemma 4.2.2: Consistency-Driven Acceleration
  • proof : Proof Sketch
  • Theorem 4.3.2: Alignment Approximation Bound
  • proof : Proof Sketch
  • proof
  • proof