TEMOS: Generating diverse human motions from textual descriptions

TL;DR

TEMOS introduces a cross-modal variational framework for generating diverse 3D human motions from natural language descriptions. By wiring text and motion through symmetric Transformer encoders into a shared latent space and decoding non-autoregressively from a single latent vector $z \in \mathbb{R}^d$ with $d=256$, TEMOS can sample multiple plausible motions per description, addressing linguistic ambiguities. The model supports both skeleton-based and SMPL body representations and is trained with reconstruction, KL, and cross-modal embedding losses, achieving state-of-the-art results on the KIT Motion-Language benchmark and demonstrating qualitative diversity and realism. The work also demonstrates SMPL-based, skinned motion synthesis, with ablations highlighting the primacy of the Transformer architecture and the utility of variational sampling; limitations include vocabulary scope and duration scalability, suggesting avenues for future research in physics-based contacts and longer sequences.

Abstract

We address the problem of generating diverse 3D human motions from textual descriptions. This challenging task requires joint modeling of both modalities: understanding and extracting useful human-centric information from the text, and then generating plausible and realistic sequences of human poses. In contrast to most previous work which focuses on generating a single, deterministic, motion from a textual description, we design a variational approach that can produce multiple diverse human motions. We propose TEMOS, a text-conditioned generative model leveraging variational autoencoder (VAE) training with human motion data, in combination with a text encoder that produces distribution parameters compatible with the VAE latent space. We show the TEMOS framework can produce both skeleton-based animations as in prior work, as well more expressive SMPL body motions. We evaluate our approach on the KIT Motion-Language benchmark and, despite being relatively straightforward, demonstrate significant improvements over the state of the art. Code and models are available on our webpage.

Figures (7)

• Figure 1: Goal: Text-to-Motions (TEMOS) learns to synthesize human motion sequences conditioned on a textual description and a duration. SMPL pose sequences are generated by sampling from a single latent vector, $z$. Here, we illustrate the diversity of our motions on two sample texts, providing three generations per text input. Each image corresponds to a motion sequence where we visualize the root trajectory projected on the ground plane and the human poses at multiple equidistant time frames. The flow of time is shown with a color code where lighter blue denotes the past.
• Figure 2: Method overview: During training, we encode both the motion and text through their respective Transformer encoders, together with modal-specific learnable distribution tokens. The encoder outputs corresponding to these tokens provide Gaussian distribution parameters on which the KL losses are applied and a latent vector $z$ is sampled. Reconstruction losses on the motion decoder outputs further provide supervision for both motion and text branches. In practice, our word embedding consists of a variational encoder that takes input from a pre-trained and frozen DistilBERT distilbert_sanh model. Trainable layers are denoted in green, the inputs/outputs in brown. At test time, we only use the right branch, which goes from an input text to a diverse set of motions through the random sampling of the latent vector $z^T$ on the cross-modal space. The output motion duration is determined by the number of positional encodings $F$.
• Figure 3: Perceptual study: (a) We ask users which motion corresponds better to the input text between two displayed samples generated from model A vs model B. (b) We ask other users which motion is more realistic without showing the textual description. We report the percentage for which the users show a preference for A. The red dashed line denotes the 50% level (equal preference). On the left of both studies, our generations from TEMOS were rated better than the previous work of Lin et al. lin2018, JL2P Ahuja2019Language2PoseNL, and Ghosh et al. Ghosh_2021_ICCV. On the right of both studies, we compare against the ground truth (GT) and see that our motions are rated as better than the GT 15.5% and 38.5% of the time, whereas Ghosh et al. Ghosh_2021_ICCV are at 8.5% and 5.5%.
• Figure 4: Qualitative comparison to the state of the art: We qualitatively compare the generations from our TEMOS model with the recent state-of-the-art methods and the ground truth (GT). We present different textual queries in columns, and different methods in rows. Overall, our generations better match semantically to the textual descriptions. We further overcome several limitations with the prior work, such as over-smooth motions in Lin et al. lin2018, foot sliding in J2LP Ahuja2019Language2PoseNL, and exaggerated foot contacts in Ghosh et al. Ghosh_2021_ICCV, which can better be viewed in our supplementary video projectpage_temos.
• Figure 5: Qualitative evaluation of the diversity: We display two motion generations for each description. Our model shows certain diversity among different generations while respecting the textual description.