DartControl: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Control

TL;DR

DART introduces a diffusion-based autoregressive framework that models long, text-driven motion as a sequence of overlapping motion primitives learned in a compact latent space. A motion-primitive VAE and latent diffusion model conditioned on history and CLIP-encoded text enable real-time, online generation with high frame rates, outperforming offline baselines in realism and speed. Spatial control is achieved via latent-space optimization and RL-based policies, enabling tasks like in-between motion, human-scene interaction, and goal-reaching, with strong results on BABEL and HML3D datasets. The approach advances open problems in text-to-motion synthesis by combining real-time generation, semantic alignment, and flexible spatial constraints, while noting limitations in vocabulary coverage and dataset dependencies.

Abstract

Text-conditioned human motion generation, which allows for user interaction through natural language, has become increasingly popular. Existing methods typically generate short, isolated motions based on a single input sentence. However, human motions are continuous and can extend over long periods, carrying rich semantics. Creating long, complex motions that precisely respond to streams of text descriptions, particularly in an online and real-time setting, remains a significant challenge. Furthermore, incorporating spatial constraints into text-conditioned motion generation presents additional challenges, as it requires aligning the motion semantics specified by text descriptions with geometric information, such as goal locations and 3D scene geometry. To address these limitations, we propose DartControl, in short DART, a Diffusion-based Autoregressive motion primitive model for Real-time Text-driven motion control. Our model effectively learns a compact motion primitive space jointly conditioned on motion history and text inputs using latent diffusion models. By autoregressively generating motion primitives based on the preceding history and current text input, DART enables real-time, sequential motion generation driven by natural language descriptions. Additionally, the learned motion primitive space allows for precise spatial motion control, which we formulate either as a latent noise optimization problem or as a Markov decision process addressed through reinforcement learning. We present effective algorithms for both approaches, demonstrating our model's versatility and superior performance in various motion synthesis tasks. Experiments show our method outperforms existing baselines in motion realism, efficiency, and controllability. Video results are available on the project page: https://zkf1997.github.io/DART/.

Figures (5)

• Figure 1: Architecture illustration of DART. The encoder network compresses the future frames $\mathbf{X}=[\mathbf{x}^{1},...,\mathbf{x}^{F}]$ into a latent variable, conditioned on the history frames $\mathbf{H}=[\mathbf{h}^{1},...,\mathbf{h}^{H}]$. The decoder network reconstructs the future frames conditioned on the history frames and the latent sample. The denoiser network predicts the clean latent sample $\hat{\mathbf{z}}_0$ conditioned on the noising step, text prompt, history frames, and noised latent sample $\mathbf{z}_t$. During the denoiser training, the encoder and decoder network weights remain fixed.
• Figure 2: Architecture of the reinforcement learning-based control policy. The pretrained DART diffusion denoiser and decoder models transform the latent actions into motion frames. The last predicted frames are canonicalized and provided to the policy model as the next step history condition.
• Figure 3: Illustrations of human-scene interaction generation given text prompts and goal pelvis joint location (visualized as a red sphere). Best viewed in the supplementary video.
• Figure 4: Illustration of the human preference study interface for evaluating motion-text semantic alignment (top) and perceptual realism(bottom). Participants are requested to select the generation that is perceptually more realistic or better aligns with the action descriptions in subtitles (only visible in semantic preference study).
• Figure 5: We demonstrate an example of integrating DART with the physics-based motion tracking method PHC Luo2023PerpetualHC to achieve more physically plausible motions. The left image illustrates a crawling sequence generated by DART, exhibiting artifacts such as hand-floor penetration. The right image displays the physics-based motion tracking outcome applied to the raw generated sequence, which enhances joint-floor contact and resolves the hand-floor penetration issue.