Learning more with the same effort: how randomization improves the robustness of a robotic deep reinforcement learning agent

TL;DR

This work addresses the data-efficiency bottleneck in robotic deep reinforcement learning and the sim-to-real transfer challenge. It compares a Progressive Neural Network (PNN) baseline trained in a fixed virtual environment with a domain-randomized variant that varies camera pose during virtual training. A robustness benchmark evaluates how well policies trained in simulation perform under diverse viewing conditions, revealing that domain randomization increases pre-transfer robustness by roughly 25% on average and expands the reliable operating region. The findings suggest that incorporating diversity in virtual training can reduce the real-data required to achieve a given robustness level, though incorporating some real experience remains beneficial for optimal performance in real-world settings.

Abstract

The industrial application of Deep Reinforcement Learning (DRL) is frequently slowed down because of the inability to generate the experience required to train the models. Collecting data often involves considerable time and economic effort that is unaffordable in most cases. Fortunately, devices like robots can be trained with synthetic experience thanks to virtual environments. With this approach, the sample efficiency problems of artificial agents are mitigated, but another issue arises: the need for efficiently transferring the synthetic experience into the real world (sim-to-real). This paper analyzes the robustness of a state-of-the-art sim-to-real technique known as progressive neural networks (PNNs) and studies how adding diversity to the synthetic experience can complement it. To better understand the drivers that lead to a lack of robustness, the robotic agent is still tested in a virtual environment to ensure total control on the divergence between the simulated and real models. The results show that a PNN-like agent exhibits a substantial decrease in its robustness at the beginning of the real training phase. Randomizing certain variables during simulation-based training significantly mitigates this issue. On average, the increase in the model's accuracy is around 25% when diversity is introduced in the training process. This improvement can be translated into a decrease in the required real experience for the same final robustness performance. Notwithstanding, adding real experience to agents should still be beneficial regardless of the quality of the virtual experience fed into the agent.

Figures (12)

• Figure 1: Interaction between the agent and the environment in an RL setting.
• Figure 2: A3C general architecture diagram. Each worker trains and evaluates their networks individually until a certain number of steps in which the parameters are shared to the global network to update it with the ones that result in better outcomes.
• Figure 3: Random initial positions of the target and joints being the camera located at $(180^{\circ}, -30^{\circ})$. Axes in the three pictures indicate pixel coordinates.
• Figure 4: Schematic diagram of the methodology.
• Figure 5: The virtual test bench grid designed to evaluate the models. The black dot refers to the camera's position in the BM training and the grey area to the camera orientations presented to the DR agent. The grid granularity is $5^\circ$.