LUCAS: Layered Universal Codec Avatars

TL;DR

LUCAS tackles the challenge of photorealistic 3D head avatars by disentangling face and hair into a layered mesh-based universal prior model that supports cross-identity generalization and real-time rendering. The approach combines a mesh-based UPM (uPiCA) with a layered face-hair architecture and a separate Gaussian rendering pathway, enabling independent deformation of hair and facial geometry and improved anchor geometry for rendering. Empirical results show improved hair fidelity, dynamic expression transfer, and zero-shot driving performance over state-of-the-art personalized and universal baselines, including better long-hair maintenance during head motion. Limitations include extreme hair deformations and unseen poses, with future work focusing on relighting, broader hairstyle coverage, and real-world fine-tuning to enhance applicability.

Abstract

Photorealistic 3D head avatar reconstruction faces critical challenges in modeling dynamic face-hair interactions and achieving cross-identity generalization, particularly during expressions and head movements. We present LUCAS, a novel Universal Prior Model (UPM) for codec avatar modeling that disentangles face and hair through a layered representation. Unlike previous UPMs that treat hair as an integral part of the head, our approach separates the modeling of the hairless head and hair into distinct branches. LUCAS is the first to introduce a mesh-based UPM, facilitating real-time rendering on devices. Our layered representation also improves the anchor geometry for precise and visually appealing Gaussian renderings. Experimental results indicate that LUCAS outperforms existing single-mesh and Gaussian-based avatar models in both quantitative and qualitative assessments, including evaluations on held-out subjects in zero-shot driving scenarios. LUCAS demonstrates superior dynamic performance in managing head pose changes, expression transfer, and hairstyle variations, thereby advancing the state-of-the-art in 3D head avatar reconstruction.

Figures (9)

• Figure 1: LUCAS: A novel approach for high-fidelity Layered Universal Codec Avatars. We disentangle face and hair into a layered structure, supporting both real-time mesh-based avatar (45 FPS on mobile) and high-fidelity Gaussian avatar generation. Our universal layered prior model also enables accurate expression and pose transfer, even for unseen subjects, while maintaining visual quality.
• Figure 2: Layered representation enables adaptive alignment between face and hair. LUCAS's independent face and hair deformation captures subtle hair movements in response to facial expressions, unlike single-mesh avatars that are globally controlled.
• Figure 3: Dehaired Head and Hair Geometries. Our method precisely disentangles dehaired head from hair for different users.
• Figure 4: Overview of LUCAS. (a) Our identity-conditioned hypernetwork $\mathcal{E}_\text{id}^\text{face}$/$\mathcal{E}_\text{id}^\text{hair}$ generates identity-specific features $\{f, d\}$ and untied biases $\Theta_\text{id}$ from neutral geometry and appearance data. (b) The expression encoder $\mathcal{E}_\text{exp}$ learns a unified expression code space that enables consistent expression transfer across identities. (c) Given expression code $z$, view direction $\omega$, and poses $\{h, \eta\}$, our compositional avatar decoder $\mathcal{D}^\text{face}$/$\mathcal{D}^\text{hair}$ produces separate geometry and appearance maps for face and hair. These are combined with mean geometry and geometry displacement for multi-mesh rendering, followed by separate pixel decoders for the final avatar image generation.
• Figure 5: Qualitative comparison (mesh). Our layered representation enables better reconstruction of long hair compared to uPiCA's single-mesh approach. While uPiCA struggles with hair-shoulder intersections and loses hair tail details during head movement, our method maintains clean geometry with accurate hair shape and positioning across different head poses.