Pose-Guided Residual Refinement for Interpretable Text-to-Motion Generation and Editing

TL;DR

The paper tackles the challenge of generating and editing 3D human motion from text while preserving structure and adding temporal detail. It introduces Pose-Guided Residual Refinement for Motion (PGR²M), which combines interpretable pose codes with residual vector quantization to capture fine-grained temporal dynamics, aided by residual dropout to preserve editability. A Base Transformer predicts pose codes from text, and a Refine Transformer predicts residual codes conditioned on text, pose codes, and stage index, enabling high-fidelity generation and precise edits. Experiments on HumanML3D and KIT-ML show improvements in FID and text-motion alignment, and user studies confirm intuitive, structure-preserving edits, highlighting the method’s practical impact for interpretable, controllable motion synthesis.

Abstract

Text-based 3D motion generation aims to automatically synthesize diverse motions from natural-language descriptions to extend user creativity, whereas motion editing modifies an existing motion sequence in response to text while preserving its overall structure. Pose-code-based frameworks such as CoMo map quantifiable pose attributes into discrete pose codes that support interpretable motion control, but their frame-wise representation struggles to capture subtle temporal dynamics and high-frequency details, often degrading reconstruction fidelity and local controllability. To address this limitation, we introduce pose-guided residual refinement for motion (PGR$^2$M), a hybrid representation that augments interpretable pose codes with residual codes learned via residual vector quantization (RVQ). A pose-guided RVQ tokenizer decomposes motion into pose latents that encode coarse global structure and residual latents that model fine-grained temporal variations. Residual dropout further discourages over-reliance on residuals, preserving the semantic alignment and editability of the pose codes. On top of this tokenizer, a base Transformer autoregressively predicts pose codes from text, and a refine Transformer predicts residual codes conditioned on text, pose codes, and quantization stage. Experiments on HumanML3D and KIT-ML show that PGR$^2$M improves Fréchet inception distance and reconstruction metrics for both generation and editing compared with CoMo and recent diffusion- and tokenization-based baselines, while user studies confirm that it enables intuitive, structure-preserving motion edits.

Figures (6)

• Figure 1: Architecture of the proposed pose-guided RVQ tokenizer. The pose code encoder produces interpretable pose latents. The 1D CNN motion encoder produces continuous motion latents. Their difference is progressively quantized by RVQ to obtain residual codes, and residual dropout stochastically masks residuals during training.
• Figure 2: Architecture of the pose code encoder. Each frame is mapped to a multi-hot vector over pose categories by the pose parser, and the pose latent is formed by a linear combination of activated pose codes.
• Figure 3: Attention-based quantization process. Queries from the input residual and keys from the codebook entries yield soft and hard selections, producing quantized residuals that are used for reconstruction and for stable training.
• Figure 4: Training procedures of the Base Transformer and the Refine Transformer. The Base Transformer autoregressively predicts pose codes conditioned on text, while the Refine Transformer predicts residual codes stage-by-stage conditioned on pose codes, text, and previously predicted residual codes.
• Figure 5: Qualitative comparison of motion generation. (a) Prompt: "the soccer player kicks the ball." (b) Prompt: "a person runs forward then abruptly turns to the left and continues running."