Stratified Avatar Generation from Sparse Observations

TL;DR

This paper tackles reconstructing 3D full-body avatars from sparse HMD observations by introducing Stratified Avatar Generation (SAGE). It decomposes the full-body motion into upper- and lower-body halves connected by a shared root and learns disentangled latent representations via separate VQ-VAE encoders/decoders, followed by a cascaded latent diffusion process that first models the upper half from sparse signals and then the lower half conditioned on the upper half. A full-body decoder fuses the two latents, with a Refiner GRU smoothing the sequence for realism. Evaluations on AMASS show state-of-the-art performance, particularly in lower-body reconstruction, demonstrating that stratified, diffusion-based generation from sparse inputs can yield plausible, immersive full-body avatars for AR/VR applications.

Abstract

Estimating 3D full-body avatars from AR/VR devices is essential for creating immersive experiences in AR/VR applications. This task is challenging due to the limited input from Head Mounted Devices, which capture only sparse observations from the head and hands. Predicting the full-body avatars, particularly the lower body, from these sparse observations presents significant difficulties. In this paper, we are inspired by the inherent property of the kinematic tree defined in the Skinned Multi-Person Linear (SMPL) model, where the upper body and lower body share only one common ancestor node, bringing the potential of decoupled reconstruction. We propose a stratified approach to decouple the conventional full-body avatar reconstruction pipeline into two stages, with the reconstruction of the upper body first and a subsequent reconstruction of the lower body conditioned on the previous stage. To implement this straightforward idea, we leverage the latent diffusion model as a powerful probabilistic generator, and train it to follow the latent distribution of decoupled motions explored by a VQ-VAE encoder-decoder model. Extensive experiments on AMASS mocap dataset demonstrate our state-of-the-art performance in the reconstruction of full-body motions.

Figures (6)

• Figure 1: Stratified avatar generation from sparse observations. Given the sensory sparse observation of the body motion: 6-DoF poses of the head and hand marked by RGB axes in (a), our method leverages a disentangled body representation in (b) to reconstruct the upper-body conditioned on the sparse observation in (c), and lower-body conditioned on the upper-body reconstruction in (d) to accomplish the full-body reconstruction in (e).
• Figure 2: The overall architecture of our SAGE Net. It mainly contains two components: (a) Disentangled VQ-VAE for discrete human motion latent learning. To facilitate visualization, we incorporate zero rotations as padding for the lower body in the Upper VQ-VAE, and vice versa for the Lower VQ-VAE. Consequently, in the visualizations of the Upper VQ-VAE, the lower body remains in a stationary pose, whereas in the visualizations of the Lower VQ-VAE, the upper body is maintained in a T-pose. (b) The stratified diffusion model, which models the conditional distribution of the latent space for upper and lower motion. This model sequentially infers the upper and lower body latents, capturing the correlation between upper and lower motions. By employing a dedicated full-body decoder on the concatenated upper and lower latents, we can obtain full-body motion.
• Figure 3: Visualization results compared with other methods. All models are trained under setting S1.
• Figure 4: Visualization results on real data.
• Figure 5: The visualization comparison for disentanglement. The darker the red color, the greater the deviation is between the predicted result and the ground truth.