GB-DQN: Gradient Boosted DQN Models for Non-stationary Reinforcement Learning
Chang-Hwan Lee, Chanseung Lee
TL;DR
This work tackles non-stationary reinforcement learning by reframing environment drift as a sequence of residual-correction tasks. It introduces Gradient-Boosted DQN (GB-DQN), an adaptive additive ensemble that adds new DQNs to fix Bellman residuals while freezing past components to avoid catastrophic forgetting. The authors provide theoretical guarantees showing residual reduction, convergence under stationary dynamics, and stability after drift, complemented by empirical results across drifted control tasks where GB-DQN outperforms DQN and various non-stationary baselines. The approach is computationally efficient, leveraging a shared replay buffer and per-step costs similar to a single DQN, making it practical for online deployment. Overall, GB-DQN offers a robust and scalable framework for adapting value estimates in dynamic environments.
Abstract
Non-stationary environments pose a fundamental challenge for deep reinforcement learning, as changes in dynamics or rewards invalidate learned value functions and cause catastrophic forgetting. We propose \emph{Gradient-Boosted Deep Q-Networks (GB-DQN)}, an adaptive ensemble method that addresses model drift through incremental residual learning. Instead of retraining a single Q-network, GB-DQN constructs an additive ensemble in which each new learner is trained to approximate the Bellman residual of the current ensemble after drift. We provide theoretical results showing that each boosting step reduces the empirical Bellman residual and that the ensemble converges to the post-drift optimal value function under standard assumptions. Experiments across a diverse set of control tasks with controlled dynamics changes demonstrate faster recovery, improved stability, and greater robustness compared to DQN and common non-stationary baselines.