Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MAGE:A Multi-stage Avatar Generator with Sparse Observations

Fangyu Du, Yang Yang, Xuehao Gao, Hongye Hou

TL;DR

MAGE addresses the challenge of reconstructing full-body motion from sparse head-mounted device observations by introducing a multi-scale diffusion framework that generates motion progressively from coarse to fine SMPL representations. It partitions the generation into three stages corresponding to skeletons with 6, 11, and 22 joints, respectively, using conditioned diffusion to enforce global structure before refining details. The model optimizes a tri-stage loss and demonstrates state-of-the-art accuracy and continuity on AMASS, with real-time inference (4-step DDIM) suitable for AR/VR applications. This approach significantly reduces error propagation in distal joints and improves temporal stability, enabling high-fidelity avatar animation from minimal sensors.

Abstract

Inferring full-body poses from Head Mounted Devices, which capture only 3-joint observations from the head and wrists, is a challenging task with wide AR/VR applications. Previous attempts focus on learning one-stage motion mapping and thus suffer from an over-large inference space for unobserved body joint motions. This often leads to unsatisfactory lower-body predictions and poor temporal consistency, resulting in unrealistic or incoherent motion sequences. To address this, we propose a powerful Multi-stage Avatar GEnerator named MAGE that factorizes this one-stage direct motion mapping learning with a progressive prediction strategy. Specifically, given initial 3-joint motions, MAGE gradually inferring multi-scale body part poses at different abstract granularity levels, starting from a 6-part body representation and gradually refining to 22 joints. With decreasing abstract levels step by step, MAGE introduces more motion context priors from former prediction stages and thus improves realistic motion completion with richer constraint conditions and less ambiguity. Extensive experiments on large-scale datasets verify that MAGE significantly outperforms state-of-the-art methods with better accuracy and continuity.

MAGE:A Multi-stage Avatar Generator with Sparse Observations

TL;DR

MAGE addresses the challenge of reconstructing full-body motion from sparse head-mounted device observations by introducing a multi-scale diffusion framework that generates motion progressively from coarse to fine SMPL representations. It partitions the generation into three stages corresponding to skeletons with 6, 11, and 22 joints, respectively, using conditioned diffusion to enforce global structure before refining details. The model optimizes a tri-stage loss and demonstrates state-of-the-art accuracy and continuity on AMASS, with real-time inference (4-step DDIM) suitable for AR/VR applications. This approach significantly reduces error propagation in distal joints and improves temporal stability, enabling high-fidelity avatar animation from minimal sensors.

Abstract

Inferring full-body poses from Head Mounted Devices, which capture only 3-joint observations from the head and wrists, is a challenging task with wide AR/VR applications. Previous attempts focus on learning one-stage motion mapping and thus suffer from an over-large inference space for unobserved body joint motions. This often leads to unsatisfactory lower-body predictions and poor temporal consistency, resulting in unrealistic or incoherent motion sequences. To address this, we propose a powerful Multi-stage Avatar GEnerator named MAGE that factorizes this one-stage direct motion mapping learning with a progressive prediction strategy. Specifically, given initial 3-joint motions, MAGE gradually inferring multi-scale body part poses at different abstract granularity levels, starting from a 6-part body representation and gradually refining to 22 joints. With decreasing abstract levels step by step, MAGE introduces more motion context priors from former prediction stages and thus improves realistic motion completion with richer constraint conditions and less ambiguity. Extensive experiments on large-scale datasets verify that MAGE significantly outperforms state-of-the-art methods with better accuracy and continuity.
Paper Structure (14 sections, 9 equations, 4 figures, 5 tables)

This paper contains 14 sections, 9 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Motion Tracking from Sparse Inputs
  4. Diffusion Models and Human Motion Generation
  5. Method
  6. Problem Formulation
  7. Multi-scale Human Motion Framework
  8. Multi-stage Diffusion Model
  9. Experiments
  10. Implementation Details
  11. Dataset and Evaluation Metrics
  12. Quantitative and Visualized Results
  13. Ablation Study
  14. Conclusion

Figures (4)

  • Figure 1: Generating full-body motion from HMDs' observations. The RGB axes represent the motion information of the head and both wrists, serving as the input to our model for generating full-body motion sequences.
  • Figure 2: Three body scales from coarse to fine. $\mathbf{S}_1$, $\mathbf{S}_2$, $\mathbf{S}_3$ contain 6, 11 composite nodes and 22 joint nodes, respectively.
  • Figure 3: The overall structure of MAGE. We utilize a sparse observation sequence and a full-body motion sequence with t steps of noise addition as inputs to the model. MAGE sequentially generates multiscale full-body motion sequences $\hat{\mathbf{S}}^{1\colon N}_1$, $\hat{\mathbf{S}}^{1\colon N}_2$, and $\hat{\mathbf{S}}^{1\colon N}_3$, where earlier stages' outputs can guide and constrain the training of subsequent stages.
  • Figure 4: Visualization results of typical motions compared with other methods under $\text{D}_1$. From top to bottom: backward walking, freestyle swimming, ballet dancing, and kicking.