Do You Have Freestyle? Expressive Humanoid Locomotion via Audio Control

TL;DR

RoboPerform addresses the challenge of expressive humanoid locomotion driven directly by audio by reframing motion as content plus style. It avoids explicit motion reconstruction by introducing a two-stage paradigm: a $ ext{Delta}$MoE teacher to learn diverse motion regimes and a diffusion-based student that injects audio style into a fixed motion content latent, enabling retargeting-free, low-latency generation. An audio-motion adaptor trained with InfoNCE aligns audio latents with motion content, facilitating rhythmically coherent actions for music-to-dance and speech-to-gesture tasks. Through extensive simulation and real-world deployment on a Unitree G1, RoboPerform demonstrates high physical plausibility, accurate audio alignment, and robust generalization across simulators and unseen audio, advancing practical audience-ready expressive humanoid control. The work introduces a new paradigm for locomotion control where motion is directly synthesized from audio conditioning, with Delta MoE enabling diverse, complementary motor styles and the diffusion policy delivering temporally precise, rhythm-aware performance.

Abstract

Humans intuitively move to sound, but current humanoid robots lack expressive improvisational capabilities, confined to predefined motions or sparse commands. Generating motion from audio and then retargeting it to robots relies on explicit motion reconstruction, leading to cascaded errors, high latency, and disjointed acoustic-actuation mapping. We propose RoboPerform, the first unified audio-to-locomotion framework that can directly generate music-driven dance and speech-driven co-speech gestures from audio. Guided by the core principle of "motion = content + style", the framework treats audio as implicit style signals and eliminates the need for explicit motion reconstruction. RoboPerform integrates a ResMoE teacher policy for adapting to diverse motion patterns and a diffusion-based student policy for audio style injection. This retargeting-free design ensures low latency and high fidelity. Experimental validation shows that RoboPerform achieves promising results in physical plausibility and audio alignment, successfully transforming robots into responsive performers capable of reacting to audio.

Figures (11)

• Figure 1: RoboPerform makes humanoid perform as dancer and talker， which utilizes audio as signal to control humanoid locomotion, enabling poolicy to generate rhythm-aligned co-speech gestures and dance movements via input speech or music.
• Figure 2: Overview of RoboPerform. We propose a two-stage approach: train an adaptor to inject kinematic information into audio modality, then a $\Delta$MoE teacher policy is trained with RL and a diffusion-based student policy is trained to denoise actions conditioned on audio latent. We propose that motion=content+style. Thus, we fix the motion latent as a constant condition and leverage different audio signals as style modulation signals to generate actions adaptive to diverse rhythms.
• Figure 3: Overview of $\Delta$MoE.
• Figure 4: T-SNE visualization results of each component for $\Delta$MoE and vanilla MoE.
• Figure 5: Ablation study on tracking performance of music-to-dance and speech-to-gesture tasks in IsaacGym and MuJoCo. The baseline uses pretrained motion generators for each task to generate motions, which drive the student policy for action generation.