UMA: Ultra-detailed Human Avatars via Multi-level Surface Alignment

TL;DR

UMA tackles the challenge of ultra-detailed animatable clothed human avatars by addressing depth misalignment and surface drift through a latent per-frame conditioning of a drivable template mesh and a multi-level surface alignment strategy grounded in avatar-guided 2D point tracking. It combines a Gaussian-based representation with vertex- and texel-level supervision, augmented by a lightweight texel super-resolution module, to recover fine garment wrinkles and textures from 6K multi-view data. The approach is validated on a new 6K-resolution, multi-view dataset and shown to achieve superior rendering quality and geometric accuracy over prior methods, with strong generalization to novel views and poses. Practical applications include VR telepresence, motion editing, and motion retargeting, highlighting UMA’s potential impact on immersive media and telepresence.

Abstract

Learning an animatable and clothed human avatar model with vivid dynamics and photorealistic appearance from multi-view videos is an important foundational research problem in computer graphics and vision. Fueled by recent advances in implicit representations, the quality of the animatable avatars has achieved an unprecedented level by attaching the implicit representation to drivable human template meshes. However, they usually fail to preserve the highest level of detail, particularly apparent when the virtual camera is zoomed in and when rendering at 4K resolution and higher. We argue that this limitation stems from inaccurate surface tracking, specifically, depth misalignment and surface drift between character geometry and the ground truth surface, which forces the detailed appearance model to compensate for geometric errors. To address this, we propose a latent deformation model and supervising the 3D deformation of the animatable character using guidance from foundational 2D video point trackers, which offer improved robustness to shading and surface variations, and are less prone to local minima than differentiable rendering. To mitigate the drift over time and lack of 3D awareness of 2D point trackers, we introduce a cascaded training strategy that generates consistent 3D point tracks by anchoring point tracks to the rendered avatar, which ultimately supervises our avatar at the vertex and texel level. To validate the effectiveness of our approach, we introduce a novel dataset comprising five multi-view video sequences, each over 10 minutes in duration, captured using 40 calibrated 6K-resolution cameras, featuring subjects dressed in clothing with challenging texture patterns and wrinkle deformations. Our approach demonstrates significantly improved performance in rendering quality and geometric accuracy over the prior state of the art.

Figures (15)

• Figure 1: Overview.UMA, takes skeletal motion and the camera view as input and generates high-fidelity geometry and appearance. To enhance the fidelity of the reconstructed human appearance and geometry, we tackle the problem from two key perspectives: avatar representation and multi-level surface alignment. For avatar representation, to address the stochasticity of the clothing dynamics that cannot be modeled by the skeletal motions, we inject a learnable latent code $\mathbf{z}_f$(zero latent $\mathbf{z}_{0}$ for testing) the drivable template $\mathbf{V}_f$ (Sec. \ref{['subsec:surface']}). A texel super resolution module $\mathcal{E}_\mathrm{sr}$ is adopted to densify the animatable gaussian textures(Sec. \ref{['subsec:srtexel']}). For multi-level surface alignment, we supervise the surface geometry at both the vertex (Sec. \ref{['subsec:triangle']}) and texel levels (Sec. \ref{['subsec:texel']}) using novel supervision derived from a foundational 2D point tracker. Specifically, the 2D point tracks $\mathbf{P}_{f,c,i}$ between the rasterized and ground-truth images obtained from the tracker are lifted and aggregated into 3D correspondences $\tilde{\mathbf{P}}_{f,i}$ across multiple views using the drivable template $\mathbf{V}_f$.
• Figure 2: Key Technical Challenges. (a) The depth misalignment leads to gradient conflicts when supervised with multi-view images, like the blue splat. (b) Even if the depth misalignment is resolved, the surface drift between different frames results in gradient conflicts, resulting in averaged and blurry appearance.
• Figure 3: Qualitative Rendering Results.UMA performs well on both novel view and novel pose synthesis tasks, and manages to capture ultra details on human avatars, i.e texture patterns, cloth wrinkles. Please zoom-in to better observe the details.
• Figure 4: Qualitative Geometry Results. For both training motions and testing motions unseen during taring, UMA generates clothing with realistic dynamics and vivid detailed deformations. Please zoom-in to better observe the details.
• Figure 5: Qualitative Rendering Comparison. We compare our approach with the competing approaches on novel view synthesis and novel pose generation. Compared with other methods, our approach preservers the best levels of details. Please zoom-in to better observe the details. We refer to the supplemental document and video for additional qualitative comparisons with more methods and for the dynamic results.