Impact of Static Friction on Sim2Real in Robotic Reinforcement Learning

TL;DR

This work identifies Static friction as a pivotal factor in the Sim2Real transfer gap for legged robotic reinforcement learning. By combining a control-theoretic joint model with parameter identification and three transfer strategies—conventional domain randomization without Static friction, Actuator Net, and Static friction–aware domain randomization—the authors demonstrate that modeling Static friction significantly improves real-world transfer, particularly for stair climbing, at the cost of increased training complexity. The study validates these findings through Sim2Sim and Sim2Real experiments using the Saturn Lite hexapod and the Rapid Motor Adaptation (RMA) algorithm, highlighting the importance of friction-aware training and proposing strategies to mitigate wear-related variability. Overall, the results offer a practical pathway to more robust Sim2Real transfer in robotic locomotion by explicitly accounting for Static friction in model randomized training and by considering joint designs with favorable friction-to-torque characteristics.

Abstract

In robotic reinforcement learning, the Sim2Real gap remains a critical challenge. However, the impact of Static friction on Sim2Real has been underexplored. Conventional domain randomization methods typically exclude Static friction from their parameter space. In our robotic reinforcement learning task, such conventional domain randomization approaches resulted in significantly underperforming real-world models. To address this Sim2Real challenge, we employed Actuator Net as an alternative to conventional domain randomization. While this method enabled successful transfer to flat-ground locomotion, it failed on complex terrains like stairs. To further investigate physical parameters affecting Sim2Real in robotic joints, we developed a control-theoretic joint model and performed systematic parameter identification. Our analysis revealed unexpectedly high friction-torque ratios in our robotic joints. To mitigate its impact, we implemented Static friction-aware domain randomization for Sim2Real. Recognizing the increased training difficulty introduced by friction modeling, we proposed a simple and novel solution to reduce learning complexity. To validate this approach, we conducted comprehensive Sim2Sim and Sim2Real experiments comparing three methods: conventional domain randomization (without Static friction), Actuator Net, and our Static friction-aware domain randomization. All experiments utilized the Rapid Motor Adaptation (RMA) algorithm. Results demonstrated that our method achieved superior adaptive capabilities and overall performance.

Figures (7)

• Figure 1: The Saturn Lite robot utilizing a reinforcement learning algorithm for locomotion. The figure captures the robot's adaptive process across stairs. During terrain adaptation, the robot relies solely on proprioceptive sensor (such as IMU and joint encoder) without exteroceptive sensor (e.g., LiDAR or cameras).
• Figure 2: The training process for the teacher network. For simplicity, the critic network $V(s_t)$ is not shown in the figure, but its input $s_t$ includes all observations: $s_t = \{ o_t^\text{prio}, o_t^{H}, o_t^\text{priv}, o_t^\text{exp}, e_t \}$
• Figure 3: Trot gait for hexapod robots. For comparison, the gait cycle diagram for quadruped robots is also provided.
• Figure 4: Training process of the robot in the student phase. During this phase, the student learns to mimic the actions of the teacher.
• Figure 5: Sim2Real implementation based on Actuator Net. When using the student network $f_0^{\text{stu}}$ with Actuator Net, the robot frequently fell while walking forward, necessitating the use of a tether. After collecting walking data with $f_0^{\text{stu}}$, Actuator Net $f^{\text{motor}}$ was trained using this data. A new round of teacher-student training was then conducted using $f^{\text{motor}}$, resulting in a new student network $f_1^{\text{stu}}$. Walking based on $f_1^{\text{stu}}$ is stable but limited to flat ground.