Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers

TL;DR

The paper tackles the challenge of real-time, on-device deep reinforcement learning under tight resource constraints by showing that batch-policy gradient methods fail with small replay buffers and proposing AVG, an incremental reparameterization-gradient method with entropy augmentation. AVG relies on normalization and TD-error scaling to stabilize updates without replay buffers or target networks, delivering learning performance comparable to batch methods on a range of simulated benchmarks and enabling real-robot demonstrations. The work provides theoretical foundations for RPG-based updates, extensive ablations to isolate stabilizing factors, and practical demonstrations on resource-constrained robots, highlighting the method's potential for lifelong, on-device adaptation. Overall, AVG offers a scalable path to deploy deep RL in real-world robotic systems with limited compute and memory, though it trades off sample efficiency and requires careful hyperparameter tuning.

Abstract

Modern deep policy gradient methods achieve effective performance on simulated robotic tasks, but they all require large replay buffers or expensive batch updates, or both, making them incompatible for real systems with resource-limited computers. We show that these methods fail catastrophically when limited to small replay buffers or during incremental learning, where updates only use the most recent sample without batch updates or a replay buffer. We propose a novel incremental deep policy gradient method -- Action Value Gradient (AVG) and a set of normalization and scaling techniques to address the challenges of instability in incremental learning. On robotic simulation benchmarks, we show that AVG is the only incremental method that learns effectively, often achieving final performance comparable to batch policy gradient methods. This advancement enabled us to show for the first time effective deep reinforcement learning with real robots using only incremental updates, employing a robotic manipulator and a mobile robot.

Figures (25)

• Figure 1: Impact of reducing replay buffer size on SAC, PPO, and TD3: Decreasing the replay buffer size adversely affects learning. In contrast, AVG succeeds despite learning without a replay buffer, as shown by a "buffer size" of 1 in the plots. Each data point represents the mean episodic return over the final 100K steps, averaged across 30 runs. All methods were trained for 10M timesteps.
• Figure 2: Effect of $\sigma$ on entropy of normal and squashed normal distribution
• Figure 3: AVG on Gymnasium and DeepMind Control Suite tasks. Each solid learning curve is an average of 30 independent runs. The shaded regions represent a 95% confidence interval of the bootstrap distribution. Note that SAC refers to SAC with a replay buffer size of $1M$. The corresponding dashed line represents the mean performance over the final 10K steps of training.
• Figure 4: AVG and IAC on the Visual Reacher task
• Figure 5: The gradient norm of the critic and actor networks for AVG and SAC, along with their average episodic returns. AVG$^-$ denotes AVG without any normalization or scaling applied. The solid lines represent the average, whereas the light lines represent the values for the individual runs. Note that the y-axis in the plots for actor and critic gradient norms is displayed on a logarithmic scale.

Theorems & Definitions (19)

• Theorem 1: Reparameterization Policy Gradient Theorem
• proof : Proof
• Theorem 2: Action Value Gradient Theorem
• proof : Proof
• Proposition 7: Compatible function approximation
• Lemma 8
• Theorem 9
• Lemma 10
• proof
• Lemma 11