MVP4D: Multi-View Portrait Video Diffusion for Animatable 4D Avatars

TL;DR

<3-5 sentence high-level summary> MVP4D tackles the challenge of creating photorealistic, animatable 4D avatars from a single reference image by introducing a morphable multi-view video diffusion model (MMVDM) that jointly generates hundreds of temporally synchronized views across 360 degrees. It leverages a diffusion transformer built on a spatio-temporal auto-encoder, with rich conditioning signals to enforce 3D and temporal coherence, and trains with a multi-modal curriculum to avoid needing large-scale multi-view video data. The generated multi-view videos are distilled into a 4D Gaussian-splat representation bound to a FLAME mesh, enabling real-time rendering and detailed facial/hair dynamics. Experimental results show superior temporal consistency and 3D fidelity compared with baselines, along with versatile generation modes and extensions such as speech-driven and text-to-4D avatars, highlighting the method’s practical potential for content creation and real-time avatars.

Abstract

Digital human avatars aim to simulate the dynamic appearance of humans in virtual environments, enabling immersive experiences across gaming, film, virtual reality, and more. However, the conventional process for creating and animating photorealistic human avatars is expensive and time-consuming, requiring large camera capture rigs and significant manual effort from professional 3D artists. With the advent of capable image and video generation models, recent methods enable automatic rendering of realistic animated avatars from a single casually captured reference image of a target subject. While these techniques significantly lower barriers to avatar creation and offer compelling realism, they lack constraints provided by multi-view information or an explicit 3D representation. So, image quality and realism degrade when rendered from viewpoints that deviate strongly from the reference image. Here, we build a video model that generates animatable multi-view videos of digital humans based on a single reference image and target expressions. Our model, MVP4D, is based on a state-of-the-art pre-trained video diffusion model and generates hundreds of frames simultaneously from viewpoints varying by up to 360 degrees around a target subject. We show how to distill the outputs of this model into a 4D avatar that can be rendered in real-time. Our approach significantly improves the realism, temporal consistency, and 3D consistency of generated avatars compared to previous methods.

Figures (11)

• Figure 1: Overview of MVP4D. The method takes as input one reference image that is encoded into the latent space of a variational autoencoder yang2024cogvideox. An off-the-shelf face tracker estimates a 3DMM for the reference image, from which we derive conditioning signals that describe camera pose, head pose, and expression. We associate additional conditioning signals with each input noisy latent video frame based on the desired generated viewpoints, poses, and expressions. A multi-view video diffusion transformer denoises the latent multi-view video. We then use the variational autoencoder to decode the latent video frames to a series of multi-view image frames (the resulting multi-view video), which is distilled into a 4D representation for real-time visualization.
• Figure 2: Our model supports different modes to generate multiview videos based on the reference image latent $\mathbf{z}_\text{ref}$ and treating different subsets of the latent video frames $\mathbf{Z}_\text{gen}$ as reference inputs.
• Figure 3: Self-reenactment results on the Nersemble and RenderMe-360 datasets. Each method generates multi-view videos from a single reference image (left). (A, B) We compare the outputs of MVP4D and baselines that support real-time rendering of the 4D avatar. MVP4D recovers fine details and structures of the face and hair that are not captured by other techniques (see insets). (C) CAP4D-MMDM and MVP4D-MMVDM generate views from a single reference image using a diffusion model. However, the CAP4D-MMDM output flickers because it relies on an image diffusion model. The flickering appears as high-frequency artifacts (black boxes) in the space-time visualization shown for a column of pixels in the generated videos (red lines). (D) Finally, we show 360-degree generation on the RenderMe-360 dataset. MVP4D outperforms baselines for this task, which fail or have noticeable artifacts, especially when generating the back of the head.
• Figure 4: Cross-reenactment results. Our method reconstructs challenging geometry, such as hair, significantly better than previous methods, while also modeling detailed dynamic effects such as wrinkles.
• Figure 5: Multi-view classifier-free guidance (CFG). Conventional CFG (top row, center) causes generations with extreme contrast, due to the poor-quality unconditional prediction (top row, right). Without CFG, artifacts appear (bottom row, left). With our multi-view CFG, the unconditional prediction (bottom row, right) looks plausible, leading to improved generation quality (bottom row, center).