Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FreeMotion: MoCap-Free Human Motion Synthesis with Multimodal Large Language Models

Zhikai Zhang, Yitang Li, Haofeng Huang, Mingxian Lin, Li Yi

TL;DR

This paper explores open-set human motion synthesis using natural language instructions as user control signals based on MLLMs across any motion task and environment and demonstrates the worth of mocap-free human motion synthesis aided by MLLMs.

Abstract

Human motion synthesis is a fundamental task in computer animation. Despite recent progress in this field utilizing deep learning and motion capture data, existing methods are always limited to specific motion categories, environments, and styles. This poor generalizability can be partially attributed to the difficulty and expense of collecting large-scale and high-quality motion data. At the same time, foundation models trained with internet-scale image and text data have demonstrated surprising world knowledge and reasoning ability for various downstream tasks. Utilizing these foundation models may help with human motion synthesis, which some recent works have superficially explored. However, these methods didn't fully unveil the foundation models' potential for this task and only support several simple actions and environments. In this paper, we for the first time, without any motion data, explore open-set human motion synthesis using natural language instructions as user control signals based on MLLMs across any motion task and environment. Our framework can be split into two stages: 1) sequential keyframe generation by utilizing MLLMs as a keyframe designer and animator; 2) motion filling between keyframes through interpolation and motion tracking. Our method can achieve general human motion synthesis for many downstream tasks. The promising results demonstrate the worth of mocap-free human motion synthesis aided by MLLMs and pave the way for future research.

FreeMotion: MoCap-Free Human Motion Synthesis with Multimodal Large Language Models

TL;DR

This paper explores open-set human motion synthesis using natural language instructions as user control signals based on MLLMs across any motion task and environment and demonstrates the worth of mocap-free human motion synthesis aided by MLLMs.

Abstract

Human motion synthesis is a fundamental task in computer animation. Despite recent progress in this field utilizing deep learning and motion capture data, existing methods are always limited to specific motion categories, environments, and styles. This poor generalizability can be partially attributed to the difficulty and expense of collecting large-scale and high-quality motion data. At the same time, foundation models trained with internet-scale image and text data have demonstrated surprising world knowledge and reasoning ability for various downstream tasks. Utilizing these foundation models may help with human motion synthesis, which some recent works have superficially explored. However, these methods didn't fully unveil the foundation models' potential for this task and only support several simple actions and environments. In this paper, we for the first time, without any motion data, explore open-set human motion synthesis using natural language instructions as user control signals based on MLLMs across any motion task and environment. Our framework can be split into two stages: 1) sequential keyframe generation by utilizing MLLMs as a keyframe designer and animator; 2) motion filling between keyframes through interpolation and motion tracking. Our method can achieve general human motion synthesis for many downstream tasks. The promising results demonstrate the worth of mocap-free human motion synthesis aided by MLLMs and pave the way for future research.
Paper Structure (53 sections, 3 equations, 10 figures, 14 tables)

This paper contains 53 sections, 3 equations, 10 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Foundation Models for Motion Synthesis
  4. Human Motion Synthesis
  5. Method
  6. Sequential Keyframe Generation from MLLMs
  7. Keyframe Designer.
  8. Keyframe Animator.
  9. Motion Filling through Interpolation and Motion Tracking
  10. Environment Visual Signals Extraction.
  11. CVAE-based Motion Control Policy.
  12. MLP-based World Model
  13. Inference
  14. Training Process.
  15. Tasks
  16. ...and 38 more sections

Figures (10)

  • Figure 1: Our method for the first time, without any motion data, explores open-set human motion synthesis using natural language instructions as user control signals based on MLLMs across any motion task and environment.
  • Figure 2: Overview of FreeMotion. FreeMotion adopts two specialized GPT-4V agents for sequential keyframe generation. Then we utilize interpolation and environment-aware motion tracking to fill the blank between keyframes.
  • Figure 3: Policy training and inference. We incorporate height maps as visual signals, enabling our policy and world model to be aware of diverse environmental conditions.
  • Figure 4: Motion synthesis visualization results of FreeMotion on HumanAct12. FreeMotion can synthesize realistic motions across different categories.
  • Figure 5: Motion synthesis visualization results on Olympic sports. FreeMotion can synthesize satisfactory motions even on challenging Olympic sports.
  • ...and 5 more figures