A Simulation Pipeline to Facilitate Real-World Robotic Reinforcement Learning Applications

TL;DR

The paper tackles the sim-to-real transfer problem in robotic reinforcement learning by proposing a multi-stage pipeline that starts with system identification and advances through core and high-fidelity simulations before real-world deployment, thereby progressively increasing realism. It adopts a goal-conditioned MDP formalism and SAC with HER, curriculum learning, and domain randomization to train robust policies, demonstrated on the Boston Dynamics Spot robot for surveillance navigation. Key contributions include a modular, reusable framework that integrates system identification, staged realism, and iterative debugging to reduce the reality gap and enable practical deployment. Findings from the case study show that policies trained in simplified and high-fidelity simulations can be transferred to the real robot with tolerances adapted to observed latency and inertia, underscoring the approach's practical impact for field robotics.

Abstract

Reinforcement learning (RL) has gained traction for its success in solving complex tasks for robotic applications. However, its deployment on physical robots remains challenging due to safety risks and the comparatively high costs of training. To avoid these problems, RL agents are often trained on simulators, which introduces a new problem related to the gap between simulation and reality. This paper presents an RL pipeline designed to help reduce the reality gap and facilitate developing and deploying RL policies for real-world robotic systems. The pipeline organizes the RL training process into an initial step for system identification and three training stages: core simulation training, high-fidelity simulation, and real-world deployment, each adding levels of realism to reduce the sim-to-real gap. Each training stage takes an input policy, improves it, and either passes the improved policy to the next stage or loops it back for further improvement. This iterative process continues until the policy achieves the desired performance. The pipeline's effectiveness is shown through a case study with the Boston Dynamics Spot mobile robot used in a surveillance application. The case study presents the steps taken at each pipeline stage to obtain an RL agent to control the robot's position and orientation.

Figures (8)

• Figure 1: Simplified diagram describing the components of the proposed RL pipeline. Each component is optional, and the combination of the stages depends on the problem's complexity.
• Figure 2: The proposed training pipeline involves three stages with increasing levels of complexity from left to right. Each stage is optional and can be revisited with different parameters until they pass a predefined performance criteria. Each stage receives as input a policy and outputs a modified policy, allowing incremental improvement until the final policy is achieved.
• Figure 3: Action set of the Spot robot. (a) Nominal and (b) feasible velocities with overlaid approximated velocities.
• Figure 4: The executed velocities of the real robot are approximated using a polynomial function approximator.
• Figure 5: Actor-critic network architecture