Preference Aligned Diffusion Planner for Quadrupedal Locomotion Control

TL;DR

This work tackles the challenge of robust quadrupedal locomotion when offline data is limited and reward labels are unavailable. It introduces a two-stage framework that first learns an offline diffusion planner from restricted expert data and then uses online interaction with weak preference labeling to diversify behavior and align with real-world dynamics. The approach yields superior stability and velocity tracking across multiple gaits and speeds and enables zero-shot sim-to-real transfer to Unitree Go1 robots. By avoiding ground-truth rewards and leveraging a lightweight online preference mechanism, the method offers data-efficient robustness gains with practical implications for legged locomotion control.

Abstract

Diffusion models demonstrate superior performance in capturing complex distributions from large-scale datasets, providing a promising solution for quadrupedal locomotion control. However, the robustness of the diffusion planner is inherently dependent on the diversity of the pre-collected datasets. To mitigate this issue, we propose a two-stage learning framework to enhance the capability of the diffusion planner under limited dataset (reward-agnostic). Through the offline stage, the diffusion planner learns the joint distribution of state-action sequences from expert datasets without using reward labels. Subsequently, we perform the online interaction in the simulation environment based on the trained offline planner, which significantly diversified the original behavior and thus improves the robustness. Specifically, we propose a novel weak preference labeling method without the ground-truth reward or human preferences. The proposed method exhibits superior stability and velocity tracking accuracy in pacing, trotting, and bounding gait under different speeds and can perform a zero-shot transfer to the real Unitree Go1 robots. The project website for this paper is at https://shangjaven.github.io/preference-aligned-diffusion-legged.

Figures (5)

• Figure 1: Failure Cases of the Offline Diffusion Planner under a Single-Source Limited Dataset: (a-c) Bounding Gait Example: The lift delay of the highlighted leg (red circle) deteriorates in subsequent motion sequences, ultimately causing the legged robot to fall. (d-f) Pacing Gait Example: The highlighted leg (red circle) demonstrates a higher lifting compared to the average step height, leading to severe lateral tilting.
• Figure 2: Video Frames in Simulation and Real World Experiments of the Proposed Architecture: (a-b) Trotting gait simulation and real-world test, (c-d) Pacing gait simulation and real-world test, (e-f) Bounding gait simulation and real-world test
• Figure 3: Proposed Architecture Framework Overview: (1) Generate Datasets: the offline datasets among pacing, trotting, and bounding gait are collected through the expert PPO policy in the $\textit{walk-these-ways}$ task. (2) Behavior Cloning: given a condition input $s^t$, the diffusion policy can produce a sequence of states and actions. (3) Preference Alignment: Conduct the preference alignment on the offline diffusion planner based on proposed weak preference labels. (4) Sim2Real: The refined policy is deployed on the Unitree Go1 robot.
• Figure 4: Velocity tracking result in 0.5 m/s bounding gait between models
• Figure 5: Bounding Gait Testing on Different Surfaces: (a) Cement Surface which is Rough and High Friction, (b) PVC Floor which is Smooth and Low Friction)