Multi-Resolution Generative Modeling of Human Motion from Limited Data

TL;DR

A generative model that learns to synthesize human motion from limited training sequences, providing conditional generation and blending across multiple temporal resolutions and demonstrating its ability to generate synchronized gestures from speech inputs, even with limited paired data.

Abstract

We present a generative model that learns to synthesize human motion from limited training sequences. Our framework provides conditional generation and blending across multiple temporal resolutions. The model adeptly captures human motion patterns by integrating skeletal convolution layers and a multi-scale architecture. Our model contains a set of generative and adversarial networks, along with embedding modules, each tailored for generating motions at specific frame rates while exerting control over their content and details. Notably, our approach also extends to the synthesis of co-speech gestures, demonstrating its ability to generate synchronized gestures from speech inputs, even with limited paired data. Through direct synthesis of SMPL pose parameters, our approach avoids test-time adjustments to fit human body meshes. Experimental results showcase our model's ability to achieve extensive coverage of training examples, while generating diverse motions, as indicated by local and global diversity metrics.

Figures (6)

• Figure 1: Overview of our motion synthesis architecture. The network generates an initial sequence of keyframes from a random noise vector and a control input. Subsequently, the sequence undergoes progressive upsampling in the temporal dimension until reaching the original resolution of the training sequence or sequences. At each level, encoders integrate control information into the motion, enabling the combination of different control signals at various temporal resolution levels during inference. We train our framework in an adversarial manner, employing a set of discriminators to evaluate the synthesized motions at each level.
• Figure 2: Generator Structure: At each step $i$, a neural network $S_i$ embeds a control input $s$ into parameters $\gamma_i$ and $\delta_i$ that modulate the current step's input via Feature-wise Linear Modulation (FiLM) perez2018film. The current step's input is a noise vector $z_i$ added to an upsampled version of the previous step's result $\hat{\Theta}_{i-1}$. The generator's role is to predict missing high-frequency details.
• Figure 3: We train our framework on distinct sequences which present different actions and emotions. When conditioned on one-hot encodings, our model is able to synthesize variations of the training examples. For instances, we can generate motion of a person talking with disgust (top) or walking around an object (bottom).
• Figure 4: Multi-scale motion generation framework: Our approach learns embeddings across various temporal resolutions from a set of control signals, which enables the synthesis of diverse motions that integrate components from different example sequences or inputs. For example, our model can generate a sequence of a person describing an object (condition A) while incorporating the joyful speaking motion of another sequence (condition B).
• Figure 5: Our model is trained using paired speech features and motion data, with additional unpaired speech samples to enhance generalization. The result is the synthesis of co-speech gestures that are synchronized with the input speech.