Lightweight High-Fidelity Low-Bitrate Talking Face Compression for 3D Video Conference

TL;DR

The paper tackles the challenge of delivering high-fidelity 3D talking-face representations at extremely low bitrates for real-time video conferencing. It introduces a metadata-driven framework that fuses FLAME-based parametric head modeling with 3D Gaussian Splatting (3DGS) to render faces from compact pose and expression parameters and a Gaussian-based head representation. A compact face-model compression scheme (Gaussian attributes and lightweight MLP offsets) substantially reduces storage and bandwidth while maintaining visual fidelity, achieving over 7× compression and rendering >170 fps with a compressed model as small as 0.59 MB. The approach outperforms 2D x265 LDP and NeRF-based baselines at low bitrates, enabling practical multi-user 3D conferencing with real-time performance and robustness to bandwidth constraints.

Abstract

The demand for immersive and interactive communication has driven advancements in 3D video conferencing, yet achieving high-fidelity 3D talking face representation at low bitrates remains a challenge. Traditional 2D video compression techniques fail to preserve fine-grained geometric and appearance details, while implicit neural rendering methods like NeRF suffer from prohibitive computational costs. To address these challenges, we propose a lightweight, high-fidelity, low-bitrate 3D talking face compression framework that integrates FLAME-based parametric modeling with 3DGS neural rendering. Our approach transmits only essential facial metadata in real time, enabling efficient reconstruction with a Gaussian-based head model. Additionally, we introduce a compact representation and compression scheme, including Gaussian attribute compression and MLP optimization, to enhance transmission efficiency. Experimental results demonstrate that our method achieves superior rate-distortion performance, delivering high-quality facial rendering at extremely low bitrates, making it well-suited for real-time 3D video conferencing applications.

Figures (4)

• Figure 1: Overall framework. The left panel shows the workflow of the proposed method. The encoder extracts expression parameter $\psi$ and pose parameter $\theta$ from the input image and encodes them into a bitstream, which is then decoded by the decoder to drive the neural renderer and generate the facial image. The right panel illustrates the neural renderer. The compressed facial model is first decoded into Gaussian attributes and MLP. The decoded expression parameters $\hat{\psi}$ are processed by the FLAME model and MLP to obtain Gaussian point positions and attribute offsets. Finally, Gaussian attributes are splatted from the viewpoint provided by the decoded pose parameters $\hat{\theta}$ to reconstruct the facial image.
• Figure 2: Overview of the compact 3D Gaussian representation.
• Figure 3: Performance comparison between our methods, x265 LDP, and NeRF-based method. All videos are encoded at 25 fps. In our methods, "UC" denotes an uncompressed facial model, while "C" represents a compressed facial model. "8bits" and "10bits" indicate that facial parameters are quantized using 8-bit and 10-bit precision levels, respectively.
• Figure 4: The qualitative comparison results. X265 LDP exhibits blocking artifacts, NeRF-based method produces unrealistic details in teeth and eyelids regions. Our method generates high-fidelity results under various poses at a lower bitrates.