LUMA: Low-Dimension Unified Motion Alignment with Dual-Path Anchoring for Text-to-Motion Diffusion Model

TL;DR

LUMA tackles semantic misalignment and gradient attenuation in diffusion-based text-to-motion by introducing dual-path semantic anchors that operate in temporal and frequency domains. A light MoCLIP encoder provides temporal semantic supervision, while low-frequency DCT components supply stable, orthogonal frequency cues; both are adaptively fused via timestep-aware FiLM modulation and a cosine-annealed dual-anchor loss to guide the denoising process. The approach achieves state-of-the-art FID and R-Precision on HumanML3D and KIT-ML and speeds up convergence by about 1.4×, without requiring large pretrained teachers. This yields efficient, high-fidelity text-to-motion generation with strong semantic grounding and robustness to language variation.

Abstract

While current diffusion-based models, typically built on U-Net architectures, have shown promising results on the text-to-motion generation task, they still suffer from semantic misalignment and kinematic artifacts. Through analysis, we identify severe gradient attenuation in the deep layers of the network as a key bottleneck, leading to insufficient learning of high-level features. To address this issue, we propose \textbf{LUMA} (\textit{\textbf{L}ow-dimension \textbf{U}nified \textbf{M}otion \textbf{A}lignment}), a text-to-motion diffusion model that incorporates dual-path anchoring to enhance semantic alignment. The first path incorporates a lightweight MoCLIP model trained via contrastive learning without relying on external data, offering semantic supervision in the temporal domain. The second path introduces complementary alignment signals in the frequency domain, extracted from low-frequency DCT components known for their rich semantic content. These two anchors are adaptively fused through a temporal modulation mechanism, allowing the model to progressively transition from coarse alignment to fine-grained semantic refinement throughout the denoising process. Experimental results on HumanML3D and KIT-ML demonstrate that LUMA achieves state-of-the-art performance, with FID scores of 0.035 and 0.123, respectively. Furthermore, LUMA accelerates convergence by 1.4$\times$ compared to the baseline, making it an efficient and scalable solution for high-fidelity text-to-motion generation.

Figures (9)

• Figure 1: LUMA generates high-fidelity 3D human motion from natural language input. It injects temporal semantic anchor from the lightweight MoCLIP encoder and frequency semantic anchor from low-frequency DCT directly into the diffusion backbone. This design overcomes gradient attenuation in deep layers and achieves state-of-the-art motion fidelity.
• Figure 2: Overview of the LUMA framework. Text embeddings from MoCLIP are injected into each U-Net block via cross-attention (dashed lines). Intermediate features are projected into two paths: a temporal semantic branch aligned with MoCLIP features and a frequency semantic branch aligned with DCT coefficients. Both branches are modulated by timestep embeddings via FiLM and integrated by the Motion Alignment module to guide denoising.
• Figure 3: Gradient magnitudes across network layers in LUMA. Left: baseline without DAL; Right: with DAL. Red and blue bars denote downsampling and upsampling layers. The dashed line marks the vanishing gradient threshold (1% of mean). DAL boosts gradients in deep layers, mitigating vanishing issues and enhancing learning.
• Figure 4: FID convergence curves for LUMA and the anchor-free baseline (vs. training steps and wall-clock time). The dashed line marks an FID threshold of 0.078. LUMA reaches this in 35k iterations (1.4$\times$ faster), reducing training time from 8 to 7 hours, demonstrating more efficient convergence.
• Figure 5: Visualization Comparison. We compare the visual results of LUMA with three state-of-the-art methods. In both examples, LUMA consistently produces more accurate, natural, and fine-grained motion than the other approaches.