Sim-to-Real Transfer in Deep Reinforcement Learning for Bipedal Locomotion

TL;DR

The chapter addresses sim-to-real transfer for DRL-based bipedal locomotion by diagnosing the gap between high-fidelity simulation and hardware, and proposing a strategic framework that combines model fidelity improvements (pre-training alignment and residual dynamics) with policy hardening (domain randomization, teacher-student training) and online adaptation. It surveys end-to-end and hierarchical control schemes, identifies main sources of mismatch in dynamics, contacts, sensing, and solvers, and details three practical levers to bridge the gap while emphasizing integration over any single method. Key contributions include formal offline system identification, residual dynamics learning, curriculum-guided domain randomization, and explicit/implicit online system identification to enable robust, scalable sim-to-real transfer. The practical impact lies in providing a structured, end-to-end roadmap for developing and evaluating robust sim-to-real solutions in legged robotics, with emphasis on verifiability, safety, and real-world adaptability.

Abstract

This chapter addresses the critical challenge of simulation-to-reality (sim-to-real) transfer for deep reinforcement learning (DRL) in bipedal locomotion. After contextualizing the problem within various control architectures, we dissect the ``curse of simulation'' by analyzing the primary sources of sim-to-real gap: robot dynamics, contact modeling, state estimation, and numerical solvers. Building on this diagnosis, we structure the solutions around two complementary philosophies. The first is to shrink the gap through model-centric strategies that systematically improve the simulator's physical fidelity. The second is to harden the policy, a complementary approach that uses in-simulation robustness training and post-deployment adaptation to make the policy inherently resilient to model inaccuracies. The chapter concludes by synthesizing these philosophies into a strategic framework, providing a clear roadmap for developing and evaluating robust sim-to-real solutions.

Figures (5)

• Figure 1: Comparison of model-based control and sim-to-real learning in terms of real-world performance. As task and system complexity increase, performance declines; however, sim-to-real learning (solid) maintains higher levels than analytical or model-based methods (dotted). The shaded region indicates the sim-to-real gap relative to the ideal upper bound determined by hardware, sensing, and safety constraints.
• Figure 2: Classification of DRL-based control schemes.
• Figure 3: End-to-end locomotion policy variants. (a) Residual: the policy adds a bounded correction to a nominal controller. (b) Guided: the policy is conditioned on a motion or controller reference. (c) Reference-free: the policy learns solely from task rewards. Feedback arrows indicate onboard estimates returned to the policy.
• Figure 4: Hierarchical control architectures for a bipedal robot. The central panel ("Standard Scheme") presents the canonical hierarchy: a task command is processed by an HL planner, whose output is executed by an LL controller to actuate the robot. The left panel ("Hybrid Variation") illustrates configurations in which one layer is learned while the other remains model-based (either a learned HL planner with a classical LL controller or the reverse). The right panel ("Learned Variation") implements both HL planning and LL control as learned policies arranged in a two-layer hierarchy.
• Figure 5: High-level roadmap for sim-to-real bipedal locomotion. Two levers help reduce the transfer gap: (1) shrinking the gap by improving simulator fidelity and system identification, and (2) hardening the policy through domain randomization (DR) and curriculum learning. Residual discrepancies are managed through online adaptation during deployment. Representative sources of mismatch include actuation and robot dynamics, sensing and state estimation, and contact and terrain modeling. The ultimate goal is to achieve robust, adaptive, and scalable bipedal locomotion.