Reconstruction-Anchored Diffusion Model for Text-to-Motion Generation

TL;DR

RAM introduces a Reconstruction-Anchored Diffusion Model for text-to-motion generation that tackles two core challenges: a representational gap from motion-agnostic text encoders and error propagation in diffusion denoising. It achieves this by learning a motion-centric latent space $z_m$ through a motion reconstruction branch and by aligning text latents $z_t$ to this space, coupled with self-regularization. At inference, Reconstructive Error Guidance (REG) uses past reconstruction to amplify improvements and guide sampling, significantly reducing error propagation. Empirical results on HumanML3D and KIT-ML show RAM sets new benchmarks in FID and semantic alignment, often outperforming strong VQ-VAE baselines while remaining competitive with diffusion-based methods. The approach offers practical impact for high-fidelity, semantically grounded motion synthesis from natural language prompts.

Abstract

Diffusion models have seen widespread adoption for text-driven human motion generation and related tasks due to their impressive generative capabilities and flexibility. However, current motion diffusion models face two major limitations: a representational gap caused by pre-trained text encoders that lack motion-specific information, and error propagation during the iterative denoising process. This paper introduces Reconstruction-Anchored Diffusion Model (RAM) to address these challenges. First, RAM leverages a motion latent space as intermediate supervision for text-to-motion generation. To this end, RAM co-trains a motion reconstruction branch with two key objective functions: self-regularization to enhance the discrimination of the motion space and motion-centric latent alignment to enable accurate mapping from text to the motion latent space. Second, we propose Reconstructive Error Guidance (REG), a testing-stage guidance mechanism that exploits the diffusion model's inherent self-correction ability to mitigate error propagation. At each denoising step, REG uses the motion reconstruction branch to reconstruct the previous estimate, reproducing the prior error patterns. By amplifying the residual between the current prediction and the reconstructed estimate, REG highlights the improvements in the current prediction. Extensive experiments demonstrate that RAM achieves significant improvements and state-of-the-art performance. Our code will be released.

Figures (4)

• Figure 1: At inference time, RAM first maps a textual description onto a motion-centric latent manifold and then predicts using a diffusion model. Meanwhile, it reconstructs previous estimates that contain error patterns. By contrasting these predictions, RAM uses the reconstruction as a negative reference to drive the output away from poor estimates and towards the real data manifold. Best viewed in color.
• Figure 2: Overview of RAM. During training, RAM learns a motion latent space through motion reconstruction, with self-regularization to encourage better separability between motion latents, resulting in improved semantic resolution. The text latents from the text encoder are drawn closer to corresponding motion latents through motion-centric latent alignment. At each inference step, given the last step prediction $\hat{\mathbf{x}}_{t+1,s}$ and text description, RAM first encodes them into latents $\mathbf{z}_{m,t+1}$ and $\mathbf{z}_\mathbf{t}$. Then, these latents, together with a zero vector $\varnothing$ and input noise, are fed into the diffusion model to separately obtain reconstruction, text-driven prediction, and unconditional prediction. These outputs are combined to produce the final output. Best viewed in color.
• Figure 3: Qualitative evaluation on the HumanML3D Dataset. Actions corresponding to the text are highlighted with green dashed lines, while unnatural artifacts are indicated with red dashed lines. It can be observed that baseline methods often fail to faithfully execute the entire set of actions described in the text. For example, in the second row ("a person walks up and tosses something"), most methods only execute the walking motion. Some outputs also exhibit distortions, such as unnatural drifting (in the third row, MoMask during sitting down and Salad during standing up) and error patterns (in the first row, MoMask’s hands move erratically up and down after completing the "drink" action). The fourth row presents the greatest challenge, involving four actions. Our method completes at least three, whereas others accomplish only one or two. Best viewed in color. Further comparisons can be found in the supplementary videos.
• Figure 4: User study results on the HumanML3D test set comparing RAM with state-of-the-art methods. We conducted a perceptual study to evaluate human preferences based on a holistic assessment of two key dimensions: motion quality and semantic alignment. Participants were instructed to rank the generated motions from different methods by jointly considering these factors. The results indicate that RAM achieves the best overall performance, demonstrating its superior capability in generating motions that are both realistic and semantically accurate.