Investigating Generalisation in Continuous Deep Reinforcement Learning

TL;DR

The paper addresses the critical problem of generalisation in continuous deep reinforcement learning by introducing a train/test benchmark that assesses robustness to domain shifts across multiple MuJoCo/OpenAI Gym tasks. It systematically evaluates state-of-the-art policy-gradient methods and several generalisation-enhancing techniques, revealing that training performance often fails to predict testing success and that no single method robustly handles all forms of variability. The study shows that factors such as observation, action, and environment noise, as well as shifts in dynamics, significantly impact performance, and that strategies like multi-domain learning, entropy regularisation, and smaller architectures can improve generalisation in some settings. Overall, the work provides a rigorous framework and practical starting points for developing RL agents that can operate reliably under real-world uncertainty and deployment-domain differences.

Abstract

Deep Reinforcement Learning has shown great success in a variety of control tasks. However, it is unclear how close we are to the vision of putting Deep RL into practice to solve real world problems. In particular, common practice in the field is to train policies on largely deterministic simulators and to evaluate algorithms through training performance alone, without a train/test distinction to ensure models generalise and are not overfitted. Moreover, it is not standard practice to check for generalisation under domain shift, although robustness to such system change between training and testing would be necessary for real-world Deep RL control, for example, in robotics. In this paper we study these issues by first characterising the sources of uncertainty that provide generalisation challenges in Deep RL. We then provide a new benchmark and thorough empirical evaluation of generalisation challenges for state of the art Deep RL methods. In particular, we show that, if generalisation is the goal, then common practice of evaluating algorithms based on their training performance leads to the wrong conclusions about algorithm choice. Finally, we evaluate several techniques for improving generalisation and draw conclusions about the most robust techniques to date.

Figures (5)

• Figure 1: Block diagram of a classic control system. $f$ represents the observation function, $g$ represents the actuation function, $h$ represents the transition function, and $\epsilon$ represents the noise that enters the system due to different types of uncertainties.
• Figure 2: Generalization of standard continuous control policies for Walker2d-v2. Top: Performance with varying testing noise $\sigma^{(o,u,s)}$ and environment variation $\Sigma$ scale. Bottom: Heatmaps illustrate policy performance over a grid of environmental domain-shifts. Each cell corresponds to a particular set of context parameters $\theta_{te}^h$ with training domain at $(0,0)$. Results are averaged over 12 random seeds.
• Figure 3: Change in normalised testing AUC score using various training settings compared to vanilla PPO training. (a) Impact of training with more stochastic environments. (b) Impact of architecture and algorithmic modifications. Results are averaged over all five training environments.
• Figure 4: Comparison of testing AUC score and training return during PPO learning of Walker2d-v2 task.
• Figure 5: Training return does not reflect generalisation performance. (a,b): Clear Pareto frontiers exist in testing AUC vs training return across algorithms (dots). (c): Testing AUC and training return are generally negatively correlated.