LS-GAN: Human Motion Synthesis with Latent-space GANs

TL;DR

This paper tackles the computational burden of text-conditioned human motion synthesis by introducing LS-GAN, a latent-space GAN framework. It encodes motions with a VAE and conditions on text via CLIP embeddings, then learns a conditional GAN in the latent space to map latent codes to new latent representations that decode back to motion. On the HumanML3D and HumanAct12 benchmarks, a simple LS-GAN (including Vanilla and Deep WGAN-GP variants) achieves competitive FID scores (e.g., 0.482 on text-to-motion) with substantial FLOPs reductions (~91% vs latent diffusion models), demonstrating faster training and inference without sacrificing quality. The work highlights latent-space GANs as a promising direction for efficient, high-quality motion synthesis and provides detailed analyses of latent-space structure and cross-task performance.

Abstract

Human motion synthesis conditioned on textual input has gained significant attention in recent years due to its potential applications in various domains such as gaming, film production, and virtual reality. Conditioned Motion synthesis takes a text input and outputs a 3D motion corresponding to the text. While previous works have explored motion synthesis using raw motion data and latent space representations with diffusion models, these approaches often suffer from high training and inference times. In this paper, we introduce a novel framework that utilizes Generative Adversarial Networks (GANs) in the latent space to enable faster training and inference while achieving results comparable to those of the state-of-the-art diffusion methods. We perform experiments on the HumanML3D, HumanAct12 benchmarks and demonstrate that a remarkably simple GAN in the latent space achieves a FID of 0.482 with more than 91% in FLOPs reduction compared to latent diffusion model. Our work opens up new possibilities for efficient and high-quality motion synthesis using latent space GANs.

Figures (8)

• Figure 1: Qualitative results of text-to-motion shown by LS-GAN
• Figure 2: Qualitative results of text-to-motion shown by our best model (Deep WGAN-GP)
• Figure 3: Method overview: The Generator $G$ maps the conditional input $[z,c]$ to a latent $z'$. The Discriminator $D$ learns to differentiate real $\mathcal{E}(x)$ vs. fake motion latent $z'$ . Finally at generation, we maps the learned latent $z'$ to motion sequence using the decoder $\mathcal{D}(z')$
• Figure 4: Visualization of the t-SNE results on latent codes of LS-GAN compared to MLD. We sample 30 motions for each action label
• Figure 5: Qualitative results of our method shown by Vanilla GAN