UniAct: Unified Motion Generation and Action Streaming for Humanoid Robots

TL;DR

UniAct presents a unified framework that bridges multimodal perception and whole-body humanoid control by combining a fine-tuned multimodal large language model with a causal streaming pipeline and a robust motion tracker. A shared FSQ-based discrete codebook aligns text, music, trajectories, and motions, enabling sub-500 ms latency and robust tracking, including a 19% improvement in zero-shot motion tracking. The authors also introduce UA-Net, a 20-hour multimodal humanoid motion dataset, to benchmark cross-modal instruction following across text, trajectory, and music conditioning, and validate across 1,000+ simulation trials and 100+ hours of real-world operation. Experimental results show UniAct surpasses strong baselines on multiple metrics and demonstrates cross-modal compositional control, offering a practical path toward responsive, general-purpose humanoid assistants and a reproducible research baseline for future work.

Abstract

A long-standing objective in humanoid robotics is the realization of versatile agents capable of following diverse multimodal instructions with human-level flexibility. Despite advances in humanoid control, bridging high-level multimodal perception with whole-body execution remains a significant bottleneck. Existing methods often struggle to translate heterogeneous instructions -- such as language, music, and trajectories -- into stable, real-time actions. Here we show that UniAct, a two-stage framework integrating a fine-tuned MLLM with a causal streaming pipeline, enables humanoid robots to execute multimodal instructions with sub-500 ms latency. By unifying inputs through a shared discrete codebook via FSQ, UniAct ensures cross-modal alignment while constraining motions to a physically grounded manifold. This approach yields a 19% improvement in the success rate of zero-shot tracking of imperfect reference motions. We validate UniAct on UniMoCap, our 20-hour humanoid motion benchmark, demonstrating robust generalization across diverse real-world scenarios. Our results mark a critical step toward responsive, general-purpose humanoid assistants capable of seamless interaction through unified perception and control.

Figures (11)

• Figure 1: UniAct, a unified framework for multimodal motion generation and action streaming. UniAct enables humanoid robots to interpret and execute diverse multimodal instructions---including natural language, musical rhythms, spatial trajectories, and reference motions---with high-fidelity performance. The architecture consists of three core components: (a) a fine-tuned mllm that translates heterogeneous inputs into discrete motion tokens via a shared codebook using fsq; (b) a causal decoding and streaming pipeline that ensures low-latency delivery of reference motions; and (c) a robust motion tracker that executes the generated motions while maintaining dynamic balance.
• Figure 2: Overview of UniAct and multimodal representations. (a) Humanoid motion is represented as temporal sequences of dof positions, tokenized via fsq. (b) Trajectory features are one-hot encoded based on the turning angle degree of segmented paths. (c) System architecture: the server-side mllm processes multimodal inputs (text, music, trajectory) and fsq-tokenized motions to autoregressively generate motion tokens; a causal decoder transforms tokens to continuous dof positions, which are streamed to the client and executed by the tracking controller for real-time motion synthesis.
• Figure 3: UA-Net dataset analysis. (a) Representative text descriptions of human motions from UA-Net. (b) Rendered motion frames corresponding to selected descriptions. (c) Action verb diversity comparison: we visualize the presence of 1684 common verbs on a square grid organized alphabetically (A--Z), where each cell represents a verb. UA-Net demonstrates significantly broader vocabulary coverage compared to HumanML3D humanml3d. (Vector graphics; zoom for details.)
• Figure 4: Qualitative results of UniAct across diverse instruction modalities. (a) Sequential text-to-motion: the humanoid executes a sequence of complex actions following instructions. (b) Trajectory-to-motion: the robot follows a curved path (yellow dashed line) with natural walking motions. (c) Music-to-motion: the humanoid generates dance movements synchronized to the music's rhythm. (d) Zero-shot human-to-humanoid motion transfer: retarget motions from internet videos to humanoid execution without additional training.
• Figure A1: Performance comparison between BeyondMimic and BeyondMimic + fsq (ours) under different noise scales. (left) MPJPE comparison. (right) Success rate comparison. As the noise scale progressively increases, BeyondMimic exhibits significant performance degradation, whereas BeyondMimic + fsq (ours) maintains relatively low MPJPE and high success rates even at the maximum noise scale (×8), which demonstrates that our quantization approach effectively constrains motions to feasible spaces, enhancing robustness.