Edge Delayed Deep Deterministic Policy Gradient: efficient continuous control for edge scenarios

TL;DR

The paper tackles the computational and memory constraints of deep reinforcement learning in edge settings by addressing Q-learning overestimation without heavy ensembles. It introduces EdgeD3, a single Q-function method that uses an expectile loss to regulate over- and underestimation, combined with target smoothing and delayed policy updates to stabilize training. Empirical results on MuJoCo benchmarks and real-world TurtleBot navigation tasks show EdgeD3 achieving competitive or superior performance while reducing GPU time by about $25\%$ and memory/compute by around $30\%$ relative to state-of-the-art methods. This makes edge-distributed, privacy-preserving on-device learning more feasible, with potential for online adaptation and broader deployment in resource-constrained environments.

Abstract

Deep Reinforcement Learning is gaining increasing attention thanks to its capability to learn complex policies in high-dimensional settings. Recent advancements utilize a dual-network architecture to learn optimal policies through the Q-learning algorithm. However, this approach has notable drawbacks, such as an overestimation bias that can disrupt the learning process and degrade the performance of the resulting policy. To address this, novel algorithms have been developed that mitigate overestimation bias by employing multiple Q-functions. Edge scenarios, which prioritize privacy, have recently gained prominence. In these settings, limited computational resources pose a significant challenge for complex Machine Learning approaches, making the efficiency of algorithms crucial for their performance. In this work, we introduce a novel Reinforcement Learning algorithm tailored for edge scenarios, called Edge Delayed Deep Deterministic Policy Gradient (EdgeD3). EdgeD3 enhances the Deep Deterministic Policy Gradient (DDPG) algorithm, achieving significantly improved performance with $25\%$ less Graphics Process Unit (GPU) time while maintaining the same memory usage. Additionally, EdgeD3 consistently matches or surpasses the performance of state-of-the-art methods across various benchmarks, all while using $30\%$ fewer computational resources and requiring $30\%$ less memory.

Figures (9)

• Figure 1: Third-degree polynomials learned optimizing the expectile loss for different values for $\alpha$ and $\beta$.
• Figure 2: Training progress curves for continuous control tasks in OpenAI Gym, showing the effect of different choices of $\alpha, \beta$ in EdgeDDPG. Plots and shaded areas indicate mean and standard deviation, respectively from evaluation across 10 trials. Benchmarks were performed on 10 random seeds for simulator and network initializations. Curves are smoothed uniformly for visual clarity.
• Figure 3: Training progress curves for continuous control tasks in OpenAI Gym, showing the effect of different choices of $\alpha, \beta$ in EdgeD3 compared to EdgeDDPG. Plots and shaded areas indicate mean and standard deviation, respectively, from evaluation across 10 trials. Benchmarks were performed on 10 random seeds for simulator and network initializations. Curves are smoothed uniformly for visual clarity.
• Figure 4: Comparing EdgeD3 with baselines in continuous control tasks. Plots are from 10 random seeds for simulator and network initializations, smoothed for visualization. Evaluations of Return are performed every 5000 time steps; plots show a standard deviation over 10 episodes.
• Figure 5: GPU time for one step of training loop for the different algorithms averaged over $10000$ steps, repeated over 10 different seeds. Details on the hardware can be found in \ref{['sec:resource_use']}

Theorems & Definitions (7)

• Definition 1
• Theorem 1: Monotonicity of Expectiles
• proof
• Corollary 1.1: Asymptotic convergence of Q-learning with Expectile loss
• Remark 1: Consistency and Asymptotic Normality of Empirical Expectiles
• Corollary 1.2: Monte Carlo Estimation of Expectile Function
• Remark 2