RSATalker: Realistic Socially-Aware Talking Head Generation for Multi-Turn Conversation

TL;DR

RSATalker addresses the limitations of both mesh-based and image-based talking head methods in multi-turn conversations by integrating 3D Gaussian Splatting with a socially-aware module. It introduces a speaker–listener motion generator, a realistic head avatar renderer, and a learnable social conditioning mechanism that encodes blood/ non-blood and equal/non-equal relationships into embeddings used to modulate motion and geometry. A three-stage training regime and a new RSATalker dataset with speech–mesh–video triplets enable end-to-end optimization for photorealism and socially coherent behavior. Experiments show state-of-the-art performance in realism, temporal fluency, and social relationship accuracy, with perceptual validation from a dedicated user study. The work advances VR avatar realism by enabling socially conditioned, crypsis-resistant multi-turn interactions with efficient 3DGS-based rendering.

Abstract

Talking head generation is increasingly important in virtual reality (VR), especially for social scenarios involving multi-turn conversation. Existing approaches face notable limitations: mesh-based 3D methods can model dual-person dialogue but lack realistic textures, while large-model-based 2D methods produce natural appearances but incur prohibitive computational costs. Recently, 3D Gaussian Splatting (3DGS) based methods achieve efficient and realistic rendering but remain speaker-only and ignore social relationships. We introduce RSATalker, the first framework that leverages 3DGS for realistic and socially-aware talking head generation with support for multi-turn conversation. Our method first drives mesh-based 3D facial motion from speech, then binds 3D Gaussians to mesh facets to render high-fidelity 2D avatar videos. To capture interpersonal dynamics, we propose a socially-aware module that encodes social relationships, including blood and non-blood as well as equal and unequal, into high-level embeddings through a learnable query mechanism. We design a three-stage training paradigm and construct the RSATalker dataset with speech-mesh-image triplets annotated with social relationships. Extensive experiments demonstrate that RSATalker achieves state-of-the-art performance in both realism and social awareness. The code and dataset will be released.

Figures (7)

• Figure 1: Pipeline of RSATalker. We introduce RSATalker, a novel framework that leverages 3D Gaussian Splatting (3DGS) to generate highly realistic and socially-aware talking heads, specifically designed for dynamic speaker–listener interaction scenarios.
• Figure 2: Pipeline of RSATalker. We introduce RSATalker, a novel framework that leverages 3D Gaussian Splatting (3DGS) to generate highly realistic and socially-aware talking heads, specifically designed for dynamic speaker–listener interaction scenarios.
• Figure 3: The structure of the socially-aware fusion network.
• Figure 4: The anchor–neural structure of the 3D Gaussians.
• Figure 5: In the deformable system, the motion of the mesh can drive the 3DGS to render a realistic 2D talking head.