Joint Learning of Depth and Appearance for Portrait Image Animation

TL;DR

The paper tackles the limitation of RGB-only portrait generation by introducing a diffusion-based framework that jointly learns appearance and depth for portraits. It expands the latent diffusion backbone and introduces a ReferenceNet to ensure identity-consistent RGBD outputs, enabling depth-conditioned editing, relighting, and audio-driven talking head animation with 3D-consistent results. The authors train on a blend of studio data with ground-truth geometry and in-the-wild sequences with pseudo-depth, yielding models that generalize to outdoor settings while supporting bi-directional RGBD tasks and RGBD video generation. This joint RGBD approach advances practical portrait manipulation by enabling coherent 3D-aware behavior and a broad set of downstream applications with potential for broader diffusion-based 3D-aware generation.

Abstract

2D portrait animation has experienced significant advancements in recent years. Much research has utilized the prior knowledge embedded in large generative diffusion models to enhance high-quality image manipulation. However, most methods only focus on generating RGB images as output, and the co-generation of consistent visual plus 3D output remains largely under-explored. In our work, we propose to jointly learn the visual appearance and depth simultaneously in a diffusion-based portrait image generator. Our method embraces the end-to-end diffusion paradigm and introduces a new architecture suitable for learning this conditional joint distribution, consisting of a reference network and a channel-expanded diffusion backbone. Once trained, our framework can be efficiently adapted to various downstream applications, such as facial depth-to-image and image-to-depth generation, portrait relighting, and audio-driven talking head animation with consistent 3D output.

Figures (10)

• Figure 1: The overview of the proposed pipeline. Given a reference image, our model jointly generates the appearance (RGB) and depth of the identity under various expressions and poses, by simply sampling random noise in the latent space.
• Figure 2: The detailed architecture of the building block of our extended model for portrait RGBD video generation. The model is equipped with additional attention modules to incorporate motion-related inputs.
• Figure 3: Qualitative comparisons with the state-of-the-art methods for monocular depth estimation on studio images.
• Figure 4: Qualitative comparisons with the state-of-the-art methods for monocular depth estimation on wild faces. Note that even 3D facial tracking (right column) can sometimes fail. Our method can achieve a better depth due to the high-quality studio data as a subset of our training data.
• Figure 5: Depth-based face editing on shape, expression and pose.