HRM^2Avatar: High-Fidelity Real-Time Mobile Avatars from Monocular Phone Scans

TL;DR

HRM^2Avatar tackles the challenge of creating high-fidelity, animatable avatars from monocular phone scans and rendering them in real time on mobile devices. It introduces a clothed mesh-driven Gaussian representation combined with a two-stage capture protocol, pose-dependent deformation, and illumination modeling, all fed through a GPU-accelerated rendering stack featuring data rearrangement, hierarchical culling, and single-pass stereo. The approach achieves high visual fidelity and real-time performance (e.g., 2048×945 at 120 FPS on mobile and 1920×1824 per eye at 90 FPS on VR headsets) and outperforms state-of-the-art monocular methods on both texture detail and dynamic clothing rendering. This work enables accessible, on-device creation and manipulation of realistic avatars for AR/VR, gaming, and social interactions, with practical implications for consumer and enterprise applications.

Abstract

We present HRM$^2$Avatar, a framework for creating high-fidelity avatars from monocular phone scans, which can be rendered and animated in real time on mobile devices. Monocular capture with smartphones provides a low-cost alternative to studio-grade multi-camera rigs, making avatar digitization accessible to non-expert users. Reconstructing high-fidelity avatars from single-view video sequences poses challenges due to limited visual and geometric data. To address these limitations, at the data level, our method leverages two types of data captured with smartphones: static pose sequences for texture reconstruction and dynamic motion sequences for learning pose-dependent deformations and lighting changes. At the representation level, we employ a lightweight yet expressive representation to reconstruct high-fidelity digital humans from sparse monocular data. We extract garment meshes from monocular data to model clothing deformations effectively, and attach illumination-aware Gaussians to the mesh surface, enabling high-fidelity rendering and capturing pose-dependent lighting. This representation efficiently learns high-resolution and dynamic information from monocular data, enabling the creation of detailed avatars. At the rendering level, real-time performance is critical for animating high-fidelity avatars in AR/VR, social gaming, and on-device creation. Our GPU-driven rendering pipeline delivers 120 FPS on mobile devices and 90 FPS on standalone VR devices at 2K resolution, over $2.7\times$ faster than representative mobile-engine baselines. Experiments show that HRM$^2$Avatar delivers superior visual realism and real-time interactivity, outperforming state-of-the-art monocular methods.

Figures (8)

• Figure 1: Method Overview. Given the two-stage phone scans of a subject, we construct a clothed mesh-driven Gaussian avatar. Static, texture-rich images impose stringent supervision on Gaussian attributes $\mathbf{g}$, while dynamic, motion-intensive sequences prioritize optimization of deformation $\Delta \mathbf{V}^d$ and illumination $L$. Through deformation MLP, illumination MLP and GPU-driven Gaussian rendering pipeline, real-time rendering and animation of realistic avatars is achievable on mobile devices.
• Figure 2: GPU-Driven Rendering Pipeline for HRM$^2$Avatar.
• Figure 3: Qualitative comparisons on NeuMan testset.
• Figure 4: Ablation Studies. All metrics are measured on the test set of the male presented in the figure.
• Figure 5: Runtime Performance on iPhone 15 Pro Max.