ICo3D: An Interactive Conversational 3D Virtual Human

TL;DR

ICo3D addresses the challenge of delivering photorealistic, interactive 3D avatars capable of natural dialogue and synchronized facial motion. It advances head and body reconstruction with HeadGaS++ and SWinGS++, respectively, and fuses them through a robust head-body integration, including cross-setup alignment. An LLM-driven conversational pipeline (ASR → LLM → TTS) drives audio-synchronized facial expressions, enabling real-time dialogue within a VR/AR context. The system achieves real-time rendering at high frame rates and demonstrates superior performance on novel-view synthesis and lip synchronization compared to state-of-the-art baselines. This work paves the way for immersive, dialogue-enabled virtual humans across gaming, virtual assistance, and education.

Abstract

This work presents Interactive Conversational 3D Virtual Human (ICo3D), a method for generating an interactive, conversational, and photorealistic 3D human avatar. Based on multi-view captures of a subject, we create an animatable 3D face model and a dynamic 3D body model, both rendered by splatting Gaussian primitives. Once merged together, they represent a lifelike virtual human avatar suitable for real-time user interactions. We equip our avatar with an LLM for conversational ability. During conversation, the audio speech of the avatar is used as a driving signal to animate the face model, enabling precise synchronization. We describe improvements to our dynamic Gaussian models that enhance photorealism: SWinGS++ for body reconstruction and HeadGaS++ for face reconstruction, and provide as well a solution to merge the separate face and body models without artifacts. We also present a demo of the complete system, showcasing several use cases of real-time conversation with the 3D avatar. Our approach offers a fully integrated virtual avatar experience, supporting both oral and written form interactions in immersive environments. ICo3D is applicable to a wide range of fields, including gaming, virtual assistance, and personalized education, among others. Project page: https://ico3d.github.io/

Figures (10)

• Figure 1: Our method generates a photorealistic 3D full-body virtual avatar from posed multi-view images, with dynamic facial expressions driven by input audio and corresponding procedural body animation. A LLM enables the user to converse with the avatar in real-time. We refer the reader to our suppl. video to see our method in action.
• Figure 2: Overview of our conversational human pipeline ICo3D. Users interact with the avatar via text or audio queries, which are processed by a LLM to produce a textual answer, then converted to audio. The audio speech is the driving signal for the audio-driven head model HeadGaS++ (Fig. \ref{['fig:headgas']}). It also serves to determine the body dynamics through procedural body animation. 3D Gaussians are generated at each timestep by both the head and the body model SWinGS++ (Fig. \ref{['fig:swings']}), which are then integrated and rendered from novel views, producing a free viewpoint video stream synchronized with the audio speech.
• Figure 3: An overview of our method for 3D animatable head reconstruction.HeadGaS++, is an extension of our work HeadGaS dhamo2023headgas, which uses features extracted from input audio speech to drive the facial expressions.
• Figure 4: An overview our dynamic 3DGS method for body reconstruction.. SWinGS++ is an extension of our work SWinGS shaw2024swings, using temporally-local dynamic MLPs on a sliding window basis. The method is extended using a spatial-temporal encoder to help reconstruct larger or faster human movements.
• Figure 5: Cross-setup head-body integration. The audio-driven head model is trained using HeadGaS++ on the RenderMe-360 dataset 2023renderme360, while the body is trained on the same subject from the DNA-Rendering dataset cheng2023dna. Left: the RenderMe-360 head is aligned to the DNA-Rendering body using Eq. \ref{['eq:alignment']}. To reduce artifacts, we prune Gaussians from the body model as discussed in Sec. \ref{['sec:head_body_integ']}. Right: we optimize for the face colors by un-freezing the last layer of the HeadGaS++ MLP and optimizing jointly with the body model.