Towards motion from video diffusion models

TL;DR

This work proposes to synthesize human motion by deforming an SMPL-X body representation guided by Score distillation sampling (SDS) calculated using a video diffusion model, and sheds light on the potential and limitations of these models for generating diverse and plausible human motions.

Abstract

Text-conditioned video diffusion models have emerged as a powerful tool in the realm of video generation and editing. But their ability to capture the nuances of human movement remains under-explored. Indeed the ability of these models to faithfully model an array of text prompts can lead to a wide host of applications in human and character animation. In this work, we take initial steps to investigate whether these models can effectively guide the synthesis of realistic human body animations. Specifically we propose to synthesize human motion by deforming an SMPL-X body representation guided by Score distillation sampling (SDS) calculated using a video diffusion model. By analyzing the fidelity of the resulting animations, we gain insights into the extent to which we can obtain motion using publicly available text-to-video diffusion models using SDS. Our findings shed light on the potential and limitations of these models for generating diverse and plausible human motions, paving the way for further research in this exciting area.

Figures (4)

• Figure 1: Human motion sequence resembling running generated using text-to-video model. The figure illustrates that the current video models can generate realistic motion for commonly occuring human activity such as running
• Figure 2: Our study consists of two stages. Stage:1 (top) Joint rotations required to animate the character are generated using PoseField. Passed through SMPLx Layer to get the final mesh which is then rasterized using a differentiable renderer. We use a random camera and a predetermined texture. This is repeated for F frames to obtain the video. Stage:2 (bottom) Rendered video is encoded to the latent space of the diffusion model then random noise is added to the latent. Unet of the Video model is used to predict the added noise and the gradients are estimated by SDS.
• Figure 3: Our results for different motions: All results are obtained by using the model VideoCraftervideocrafter2. (i) Walking motion is one of the best cases in addition to running (Fig:\ref{['fig:teaser']}). (ii) Punching is a semi-failure case (iii) Cartwheel is an extreme failure case.
• Figure 4: Visualization of the optimized latent when given the two actions as prompts. Generated videos in the top row of each video model denote the action "running". The bottom row of each denotes the action "punching". Clearly, the top rows of each model show a more natural motion. VideoCraftervideocrafter2 demonstrates a higher degree of realism in both actions compared to other models