IM-Portrait: Learning 3D-aware Video Diffusion for Photorealistic Talking Heads from Monocular Videos

TL;DR

IM-Portrait introduces a 3D-aware diffusion framework that directly generates photoreal talking head MPIs from a single identity image and control signals, enabling efficient novel-view rendering suitable for VR. By integrating differentiable MPI rendering into the diffusion process and employing Reference-Target Alternating Training with bootstrapping, the model learns 3D structure from monocular videos without multi-view data. It achieves competitive image and superior temporal quality while enabling stereo and perspective rendering, often at faster speeds than NeRF-based approaches. The work advances practical 3D-consistent, high-fidelity talking head synthesis from readily available data, with potential impact on immersive communication and virtual avatar applications.

Abstract

We propose a novel 3D-aware diffusion-based method for generating photorealistic talking head videos directly from a single identity image and explicit control signals (e.g., expressions). Our method generates Multiplane Images (MPIs) that ensure geometric consistency, making them ideal for immersive viewing experiences like binocular videos for VR headsets. Unlike existing methods that often require a separate stage or joint optimization to reconstruct a 3D representation (such as NeRF or 3D Gaussians), our approach directly generates the final output through a single denoising process, eliminating the need for post-processing steps to render novel views efficiently. To effectively learn from monocular videos, we introduce a training mechanism that reconstructs the output MPI randomly in either the target or the reference camera space. This approach enables the model to simultaneously learn sharp image details and underlying 3D information. Extensive experiments demonstrate the effectiveness of our method, which achieves competitive avatar quality and novel-view rendering capabilities, even without explicit 3D reconstruction or high-quality multi-view training data.

Figures (14)

• Figure 1: We propose a 3D-aware video diffusion model for talking head synthesis. Given an image as identity and a sequence of tracking signals (as shown on the left for each example), our model directly generates videos in Multiplane Images (MPIs) in a single denoising process, which is ready for efficient novel-view rendering. This enables immersive viewing experience, e.g. rendering binocular stereo or perspective distortion in VR headset. Please see our website for more results: https://y-u-a-n-l-i.github.io/projects/IM-Portrait/.
• Figure 2: Inference pipeline. Our model is built on the architecture of Lumiere bar2024lumiere, which takes an identity image, 2D noise video, the sequence of expressions rendered from 3DMM model and first frame image as input and outputs MPI video sequences. During inference, the network is conditioned on a reference portrait and takes the last frame of previously generated clip as the first frame condition. We separate the network into a color branch and a geometry branch. The two branches share information by zero convolution zhang2023adding.
• Figure 3: Illustration of Reference-Target Alternating Training. On the top branch, we construct MPIs in the target camera and render the frontal view, where we directly use the ground truth target image to compute loss \ref{['eq:main_loss_0']} to learn sharp renderings and bootstrap the model. On the bottom branch, we construct MPIs in the reference camera. We first rasterize the target control signal (expression) in the reference camera, then use the bootstrapped model to generate pseudo ground truth by rendering the frontal view. The pseudo ground truth is added with noise, serving as the model input to compute loss \ref{['eq:main_loss_1']} for learning 3D shapes.
• Figure 4: Talking head results. We show results from previous work and our method. Our method generates results with sharp appearance and closely aligned expression with the ground truth. Note Portrait4d-v2 uses a customized camera space which is non-trivial to render GT camera aligned images.
• Figure 5: Binocular stereo and perspective effects. Our method generates MPI videos that enables efficient spatial rendering, e.g. binocular stereo and perspective effects. We render stereo pairs and calculate a disparity using RAFT-Stereo lipson2021raft to visualize the perceived geometry. To demonstrate perspective effects, we render a view with camera moving closer to the subject. Our method renders visually plausible and comparable 3D effects with NeRF based approach like Portrait4d-v2, while being orders of magnitude faster.