Splat-Portrait: Generalizing Talking Heads with Gaussian Splatting

TL;DR

Splat-Portrait addresses talking-head generation from a single portrait by learning pixel-aligned Gaussian splats to represent static head geometry and a predicted 2D background, while conditioning dynamic splat offsets on audio to produce lip motion. Trained in two stages without 3D supervision, it distills appearance knowledge from a strong 2D diffusion prior to improve extreme-view synthesis. The approach achieves state-of-the-art results on monocular datasets HDTF and TalkingHead-1KH, outperforming baselines in fidelity, identity preservation, and lip synchronization while remaining efficient and lightweight. By explicitly disentangling static geometry from dynamic motion and integrating background context, Splat-Portrait delivers higher-quality novel-view renderings and robust cross-identity generalization. The authors also provide public code and supplementary material for reproducibility.

Abstract

Talking Head Generation aims at synthesizing natural-looking talking videos from speech and a single portrait image. Previous 3D talking head generation methods have relied on domain-specific heuristics such as warping-based facial motion representation priors to animate talking motions, yet still produce inaccurate 3D avatar reconstructions, thus undermining the realism of generated animations. We introduce Splat-Portrait, a Gaussian-splatting-based method that addresses the challenges of 3D head reconstruction and lip motion synthesis. Our approach automatically learns to disentangle a single portrait image into a static 3D reconstruction represented as static Gaussian Splatting, and a predicted whole-image 2D background. It then generates natural lip motion conditioned on input audio, without any motion driven priors. Training is driven purely by 2D reconstruction and score-distillation losses, without 3D supervision nor landmarks. Experimental results demonstrate that Splat-Portrait exhibits superior performance on talking head generation and novel view synthesis, achieving better visual quality compared to previous works. Our project code and supplementary documents are public available at https://github.com/stonewalking/Splat-portrait.

Figures (3)

• Figure 1: Overview of Splat-Portrait. The identity image $I_i$ is passed through a U-Net Static Generator(SG) to reconstruct static 3D Gaussian Splats, alpha-blended over a predicted 2D background. The dynamic decoder estimates splat offsets at timestep $T_n$ using audio features $A_n$ and time embedding $\Delta T$. The training procedure consists of two stages, stage(I): an initial pre-training phase, where the static components are trained on a large-scale dataset using a static reconstruction loss $\mathcal{L}_{\mathrm{static}}$, and stage(II): a fine-tuning phase on a smaller dataset incorporating an additional dynamic reconstruction loss $\mathcal{L}_{\mathrm{dynamic}}$. And a score distillation loss $\mathcal{L}_{\mathrm{SDS}}$ on extreme viewpoints applied during both stages.
• Figure 2: Qualitative results. Top: We show source frames from five videos, future predicted frames from ours and baselines, and future depths from ours. Bottom: Additional examples of 3D reconstruction, for our method and Real3D-Portrait, displaying the input frame, and the reconstructed depth-map from each method.
• Figure 3: Ablation study, with extreme head yaw angles (top row at -35°, the middle row at 0°, and the bottom row at +35°).