Gaussian Head & Shoulders: High Fidelity Neural Upper Body Avatars with Anchor Gaussian Guided Texture Warping

TL;DR

This work addresses the gap in neural avatars that convincingly model clothed upper bodies from monocular video. It introduces Gaussian Head & Shoulders, which uses FLAME-driven head Gaussians and a sparse set of anchor Gaussians to constrain a pose-aware neural texture warping field, enabling high-frequency clothing details without reliance on accurate UV mapping. By combining a coarse texture, a pose-dependent latent texture, and a neural warping field, the method achieves superior self- and cross-reenactment fidelity while maintaining real-time rendering speeds; an accelerated no-MLP inference path yields approximately $130$ FPS. The approach reduces the number of Gaussians needed for detailed clothing, improves texture sharpness, and demonstrates robust performance across casual monocular videos, with practical implications for AR/VR and telepresence. Limitations include difficulty handling extreme self-occlusion, which is discussed in supplementary materials and addressed by the full (MLP-enabled) variant when necessary.

Abstract

By equipping the most recent 3D Gaussian Splatting representation with head 3D morphable models (3DMM), existing methods manage to create head avatars with high fidelity. However, most existing methods only reconstruct a head without the body, substantially limiting their application scenarios. We found that naively applying Gaussians to model the clothed chest and shoulders tends to result in blurry reconstruction and noisy floaters under novel poses. This is because of the fundamental limitation of Gaussians and point clouds -- each Gaussian or point can only have a single directional radiance without spatial variance, therefore an unnecessarily large number of them is required to represent complicated spatially varying texture, even for simple geometry. In contrast, we propose to model the body part with a neural texture that consists of coarse and pose-dependent fine colors. To properly render the body texture for each view and pose without accurate geometry nor UV mapping, we optimize another sparse set of Gaussians as anchors that constrain the neural warping field that maps image plane coordinates to the texture space. We demonstrate that Gaussian Head & Shoulders can fit the high-frequency details on the clothed upper body with high fidelity and potentially improve the accuracy and fidelity of the head region. We evaluate our method with casual phone-captured and internet videos and show our method archives superior reconstruction quality and robustness in both self and cross reenactment tasks. To fully utilize the efficient rendering speed of Gaussian splatting, we additionally propose an accelerated inference method of our trained model without Multi-Layer Perceptron (MLP) queries and reach a stable rendering speed of around 130 FPS for any subjects.

Figures (9)

• Figure 1: Method. (a) We utilize a set of standard head Gaussians and anchor Gaussians driven by LBS with the FLAME model. (b) Anchor Gaussians are initialized with a set of corresponding target coordinates in the texture space. This 3D-2D correspondence is used to constrain (c) a neural texture warping field that maps each pixel on the image plane $\mathbf{x}_v$ to a pixel in the texture space $\mathbf{x}_t$. (d) We then sample in the texture space to fetch the coarse texture $\mathbf{T}_c$ and latent texture $\mathbf{T}_f$, which is parsed by an MLP to obtain pose-dependent fine texture $\mathbf{C}_f^t$. Both coarse and fine textures are then combined to form a body texture, which is blended with other Gaussians through alpha compositing to form the final rendering.
• Figure 2: Qualitative comparison of self-reenactment task. We show that both full version and No MLP version of our method can effectively recover a more accurate and more robust body texture, even under cases of extreme poses and high-frequency cloth textures.
• Figure 3: Qualitative evaluation of cross-identity reenactment. In addition to the improvement in cloth texture quality and robustness, we found that our approach often leads to more accurate expression control. This is because we are using much fewer LBS-driven Gaussians for the body part, therefore the capacity of LBS weight inference network can fully focus on the head region.
• Figure 4: Ablation. The anchor constraint is necessary for learning sharp textures even if the body only moves slightly during the video.
• Figure 5: Ablation for cross-identity reenactment. The additional Euclidean transformation helps to align the body texture with head Gaussian under novel poses, whereas $\mathcal{L}_{warp}$ is necessary to prevent arbitrary warping of the white background.