GAT-NeRF: Geometry-Aware-Transformer Enhanced Neural Radiance Fields for High-Fidelity 4D Facial Avatars

TL;DR

This work tackles the challenge of reconstructing high-fidelity 4D facial avatars from monocular video, where traditional NeRF-based methods struggle to capture fine geometric details. It introduces GAT-NeRF, a hybrid framework that embeds a Geometry-Aware Transformer (GAT) into NeRF, fusing coordinate encodings, 3DMM expression parameters, and per-frame latent codes to enhance per-point feature representations and recover dynamic wrinkles and skin textures. The method employs a two-stage volumetric rendering pipeline with coarse and fine networks, and an end-to-end training objective that includes latent-code regularization, enabling both high-fidelity reconstruction and controllable expression/pose manipulation. Experimental results on the NeRFace dataset demonstrate state-of-the-art fidelity in L1 and SSIM while providing robust expression and pose control, highlighting the potential of geometry-guided attention to advance realistic digital humans, albeit with noted limitations in generalization and real-time efficiency that point to future hybrid-prior and speed-optimized approaches.

Abstract

High-fidelity 4D dynamic facial avatar reconstruction from monocular video is a critical yet challenging task, driven by increasing demands for immersive virtual human applications. While Neural Radiance Fields (NeRF) have advanced scene representation, their capacity to capture high-frequency facial details, such as dynamic wrinkles and subtle textures from information-constrained monocular streams, requires significant enhancement. To tackle this challenge, we propose a novel hybrid neural radiance field framework, called Geometry-Aware-Transformer Enhanced NeRF (GAT-NeRF) for high-fidelity and controllable 4D facial avatar reconstruction, which integrates the Transformer mechanism into the NeRF pipeline. GAT-NeRF synergistically combines a coordinate-aligned Multilayer Perceptron (MLP) with a lightweight Transformer module, termed as Geometry-Aware-Transformer (GAT) due to its processing of multi-modal inputs containing explicit geometric priors. The GAT module is enabled by fusing multi-modal input features, including 3D spatial coordinates, 3D Morphable Model (3DMM) expression parameters, and learnable latent codes to effectively learn and enhance feature representations pertinent to fine-grained geometry. The Transformer's effective feature learning capabilities are leveraged to significantly augment the modeling of complex local facial patterns like dynamic wrinkles and acne scars. Comprehensive experiments unequivocally demonstrate GAT-NeRF's state-of-the-art performance in visual fidelity and high-frequency detail recovery, forging new pathways for creating realistic dynamic digital humans for multimedia applications.

Figures (6)

• Figure 1: Overview of the proposed GAT-NeRF framework. For each input frame, 3D coordinates ($\mathbf{p}$) are sampled along camera rays in a canonical space. These coordinates, along with corresponding facial expression parameters ($\bm{\delta}$) derived from a 3DMM and a learnable latent code ($\bm{\gamma}$), are positionally encoded and concatenated. This fused multi-modal feature vector is then processed by our Geometry-Aware Transformer (GAT) module. The GAT leverages its self-attention mechanism and non-linear transformations to perform dynamic feature re-weighting and enhancement on these per-point, geometry-rich inputs. The resulting enhanced features then guide a subsequent MLP to predict view-dependent color ($\mathbf{c}$) and density ($\sigma$), enabling the detailed capture of high-frequency facial attributes (e.g., dynamic wrinkles and acne scars) via volume rendering.
• Figure 2: Architecture of the MLP networks following the GAT module in GAT-NeRF. The GAT output features are processed by a Backbone MLP, which incorporates a skip connection to fuse initial multi-modal inputs (positional encoding, expression, and latent codes) with intermediate features. This backbone subsequently predicts volume density ($\sigma$). The features are also used, along with view directions, by a separate Color MLP to predict RGB color ($\mathbf{c}$).
• Figure 3: We qualitatively compare our GAT-NeRF approach with classical and state-of-the-art face reconstruction methods. From left to right: NeRFace Gafni21DynamicNeural, PointAvatar Zheng23Pointavatar, FlashAvatar Xiang24Flashavatar, the proposed GAT-NeRF, and the ground-truth image. Zoomed-in regions highlight differences in fine detail preservation, particularly around the eyes, mouth, and forehead, where our method demonstrates superior reconstruction of dynamic wrinkles and subtle textures.
• Figure 4: Our dynamic neural radiance field enables decoupled control of expression vectors and head poses.
• Figure 5: Our 4D facial avatar enables facial expression reenactment, where the expressions of a source subject are transferred to a target subject represented by our dynamic neural radiance field.