Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shift

TL;DR

This work extends off-policy corrections for deep reinforcement learning by bootstrapping covariate shift through COP-TD. It introduces a discounted COP-TD rule and a soft ratio normalization to address two practical limitations: lack of contraction guarantees and projection requirements with nonlinear function approximators. The proposed methods are analyzed theoretically, with contraction guarantees for the discounted operator under certain conditions, and evaluated empirically on Atari games, showing stability and performance gains in several titles. The results highlight the value of properly accounting for distribution mismatch in offline/Replay-based learning, and point to future directions in robust ratio learning and combining with forward off-policy methods. Overall, discounted COP-TD offers a more stable and scalable path for off-policy value learning with deep networks.

Abstract

In this paper we revisit the method of off-policy corrections for reinforcement learning (COP-TD) pioneered by Hallak et al. (2017). Under this method, online updates to the value function are reweighted to avoid divergence issues typical of off-policy learning. While Hallak et al.'s solution is appealing, it cannot easily be transferred to nonlinear function approximation. First, it requires a projection step onto the probability simplex; second, even though the operator describing the expected behavior of the off-policy learning algorithm is convergent, it is not known to be a contraction mapping, and hence, may be more unstable in practice. We address these two issues by introducing a discount factor into COP-TD. We analyze the behavior of discounted COP-TD and find it better behaved from a theoretical perspective. We also propose an alternative soft normalization penalty that can be minimized online and obviates the need for an explicit projection step. We complement our analysis with an empirical evaluation of the two techniques in an off-policy setting on the game Pong from the Atari domain where we find discounted COP-TD to be better behaved in practice than the soft normalization penalty. Finally, we perform a more extensive evaluation of discounted COP-TD in 5 games of the Atari domain, where we find performance gains for our approach.

Figures (8)

• Figure 1: $\eta = 0.002$ with 5 seeds per run for 150 iterations. Left. Comparing discount factors in Pong. Using a discount factor gives a significant performance improvement. Right. Comparing normalization weights in Pong. Using normalization helps learning, but a large normalization weight causes divergence in the $c$ values.
• Figure 2: $\eta = 0.02$ with 3 seeds for 150 iterations. Performance of discounted COP-TD with a small target update period of 1000 and $\hat{\gamma}=0.99$ on 5 Atari 2600 games.
• Figure 3: $\eta = 0.002$ with 3 seeds for 50 iterations. 4-way performance comparison using the discounted COP-TD loss as an auxiliary task and TD error prioritization as in schaul16prioritized, blue line corresponds to the corrected agent with $\hat{\gamma} = 0.999$ at iteration 50.
• Figure 4: From same runs shown in Figure \ref{['fig:sweep']}, left. Average predicted ratio in evaluation episode for a set of $\hat{\gamma}$ in the game of Pong.
• Figure 5: Sample states (frames) encountered under the random policy, predicted either as relatively less likely under $\pi$ (low $c$) or relatively more likely under $\pi$ (high $c$). The experiment clipped the corrections at 0.0025 which was later found to be unnecessary.

Theorems & Definitions (36)

• Theorem 1
• Theorem 2
• Corollary 1
• Lemma 1
• Definition 1
• Definition 2
• Proposition 1
• Lemma 2
• Theorem 3
• Lemma 3