Avat3r: Large Animatable Gaussian Reconstruction Model for High-fidelity 3D Head Avatars

TL;DR

<3-5 sentence high-level summary> Avat3r tackles the challenge of producing animatable, photo-realistic 3D head avatars from a handful of images without expensive test-time optimization. It introduces a Large Reconstruction Model that outputs per-pixel 3D Gaussian primitives and uses cross-attention to an expression code to enable facial animation, guided by priors from DUSt3R and Sapiens. The method is trained with inconsistent-input robustness by leveraging multi-timestep training and 3D Gaussian splatting losses, achieving competitive results against state-of-the-art few-shot methods and showing applicability to out-of-domain inputs like antique busts. This work broadens practical access to high-quality, animatable digital heads for casual use and cross-domain applications, with potential for diffusion-based 3D lifting and expanded monocular video training.

Abstract

Traditionally, creating photo-realistic 3D head avatars requires a studio-level multi-view capture setup and expensive optimization during test-time, limiting the use of digital human doubles to the VFX industry or offline renderings. To address this shortcoming, we present Avat3r, which regresses a high-quality and animatable 3D head avatar from just a few input images, vastly reducing compute requirements during inference. More specifically, we make Large Reconstruction Models animatable and learn a powerful prior over 3D human heads from a large multi-view video dataset. For better 3D head reconstructions, we employ position maps from DUSt3R and generalized feature maps from the human foundation model Sapiens. To animate the 3D head, our key discovery is that simple cross-attention to an expression code is already sufficient. Finally, we increase robustness by feeding input images with different expressions to our model during training, enabling the reconstruction of 3D head avatars from inconsistent inputs, e.g., an imperfect phone capture with accidental movement, or frames from a monocular video. We compare Avat3r with current state-of-the-art methods for few-input and single-input scenarios, and find that our method has a competitive advantage in both tasks. Finally, we demonstrate the wide applicability of our proposed model, creating 3D head avatars from images of different sources, smartphone captures, single images, and even out-of-domain inputs like antique busts. Project website: https://tobias-kirschstein.github.io/avat3r/

Figures (17)

• Figure 1: Avat3r. Given just four images of a person's head (a), Avat3r achieves two things: (b) it creates a faithful 3D reconstruction of the head in a feed-forward manner, and (c) it allows facial animation without having seen any of the corresponding expressions of the person. This simplifies the capturing process as it drops the need for recording long sequences of facial movement. As a result, the entire pipeline from head scan to final 3D head avatar can be executed within a few minutes and runs on a single consumer-grade GPU.
• Figure 2: Method Overview: Avat3r reconstructs a high-quality 3D representation from just a few input images by predicting 3D Gaussian attributes for each input pixel. We first obtain position maps $I^{pos}$ from DUSt3R and feature maps $I^{feat}$ from Sapiens for each view. These are then patchified to serve as image tokens for the Vision Transformer back-end. Dense Self-attention performs matching within tokens of the same image and across views to infer 3D structure. Dynamics are modelled via cross-attention layers that attend to a sequenced expression code. The resulting intermediate feature maps are decoded into Gaussian position, scale, rotation, color, and opactiy maps, and then upsampled to the original input image resolution. Finally, we add skip connections to the predicted position and color maps. Gaussians that belong to pixels with a confidence larger than a pre-defined threshold are collected and can be rendered from arbitrary viewpoints.
• Figure 3: Comparison for few-shot 3D Head Avatar creation. We evaluate the ability of Avat3r to create 3D head avatars from four input images in a self-reenactment setting. Note that the NeRSemble dataset has not been used during training of Avat3r and therefore both source and driver person are out-of-domain in this case. Nevertheless, our method produces high quality avatars.
• Figure 4: Ablation study. Adding Sapiens features (b) noticeably improves sharpness but still exhibits misalignments. Conversely, using position maps from Dust3r improves alignment but is overall less sharp (c). Combining both but training on multi-view images from the same timestep produces sharp results but is less robust to inconsistencies in the input (d). Our final model exhibits the best trade-off between alignment, sharpness, and robustness (e).
• Figure 5: Expression Generalization.