LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioning

TL;DR

The paper addresses the misalignment between natural language and dynamic human motion by moving from a language-vision space to a language-motion latent space using LaMP.LaMP jointly pretrains a motion transformer and a text transformer with four proxy tasks, leveraging a VQ-VAE to discretize motion and learn motion-aware text features, plus language-aware motion features.The approach yields state-of-the-art results across text-to-motion generation, motion-text retrieval, and motion captioning on benchmarks like HumanML3D and KIT-ML, and introduces LaMP-BertScore as a semantic alignment metric.Overall, LaMP demonstrates strong cross-modal performance, practical utility for motion-driven applications, and a pathway toward robust language-grounded motion understanding.

Abstract

Language plays a vital role in the realm of human motion. Existing methods have largely depended on CLIP text embeddings for motion generation, yet they fall short in effectively aligning language and motion due to CLIP's pretraining on static image-text pairs. This work introduces LaMP, a novel Language-Motion Pretraining model, which transitions from a language-vision to a more suitable language-motion latent space. It addresses key limitations by generating motion-informative text embeddings, significantly enhancing the relevance and semantics of generated motion sequences. With LaMP, we advance three key tasks: text-to-motion generation, motion-text retrieval, and motion captioning through aligned language-motion representation learning. For generation, we utilize LaMP to provide the text condition instead of CLIP, and an autoregressive masked prediction is designed to achieve mask modeling without rank collapse in transformers. For retrieval, motion features from LaMP's motion transformer interact with query tokens to retrieve text features from the text transformer, and vice versa. For captioning, we finetune a large language model with the language-informative motion features to develop a strong motion captioning model. In addition, we introduce the LaMP-BertScore metric to assess the alignment of generated motions with textual descriptions. Extensive experimental results on multiple datasets demonstrate substantial improvements over previous methods across all three tasks. The code of our method will be made public.

Figures (12)

• Figure 1: LaMP overview. We conduct joint training for contrastive learning, matching, and bidirectional text-motion translation by leveraging the textual features extracted from tokenized text descriptions via the text transformer and the motion features derived from the motion transformer.
• Figure 2: LaMP-T2M and LaMP-M2T frameworks overview. (Left) Pretrained LaMP's text transformer is employed to extract condition embedding and autoregressive mask prediction is performed. (Right) Finetuning an LLM to achieve motion captioning.
• Figure 3: Heatmap of similarity matrix. The diagonal represents positive sample pairs, with darker colors indicating better quality.
• Figure 4: Qualitative results of text-to-motion generation on HumanML3D.
• Figure A1: Overview of VQVAE.