IDOL: Unified Dual-Modal Latent Diffusion for Human-Centric Joint Video-Depth Generation

TL;DR

IDOL tackles human-centric joint video-depth generation by introducing a unified dual-modal latent diffusion framework. A parameter-sharing unified U-Net with a modality label and cross-modal attention enables joint video and depth denoising, while motion consistency and cross-attention map consistency losses enforce precise video-depth spatial alignment. The HAOP pre-training strategy mitigates background leakage and improves appearance control. Empirical results on TikTok and NTU120 demonstrate state-of-the-art video quality and depth accuracy across different depth maps, with favorable efficiency and generalization to various pose controls. This work advances practical, controllable generation of temporally coherent 2.5D content and informs multi-modal diffusion modeling for aligned object-centric outputs.

Abstract

Significant advances have been made in human-centric video generation, yet the joint video-depth generation problem remains underexplored. Most existing monocular depth estimation methods may not generalize well to synthesized images or videos, and multi-view-based methods have difficulty controlling the human appearance and motion. In this work, we present IDOL (unIfied Dual-mOdal Latent diffusion) for high-quality human-centric joint video-depth generation. Our IDOL consists of two novel designs. First, to enable dual-modal generation and maximize the information exchange between video and depth generation, we propose a unified dual-modal U-Net, a parameter-sharing framework for joint video and depth denoising, wherein a modality label guides the denoising target, and cross-modal attention enables the mutual information flow. Second, to ensure a precise video-depth spatial alignment, we propose a motion consistency loss that enforces consistency between the video and depth feature motion fields, leading to harmonized outputs. Additionally, a cross-attention map consistency loss is applied to align the cross-attention map of the video denoising with that of the depth denoising, further facilitating spatial alignment. Extensive experiments on the TikTok and NTU120 datasets show our superior performance, significantly surpassing existing methods in terms of video FVD and depth accuracy.

Figures (9)

• Figure 1: Given a human foreground image, an arbitrary background image, and a defined pose sequence, our IDOL generates high-fidelity video and the corresponding depth maps, which can be rendered as realistic 2.5D video.
• Figure 2: Left: Overall model architecture. Our IDOL features a unified dual-modal U-Net (gray boxes), a parameter-sharing design for joint video-depth denoising, wherein the denoising target is controlled by a one-hot modality label ($y_{\text{v}}$ for video and $y_\text{d}$ for depth). Right: Individual U-Net block structure. Cross-modal attention is added to enable mutual information flow between video and depth features, with consistency loss terms $\mathcal{L}_{\text{mo}}$ and $\mathcal{L}_{\text{xattn}}$ ensuring the video-depth alignment. Skip connections are omitted for conciseness.
• Figure 2: Quantitative comparison between our method and existing methods on the TikTok and NTU120 datasets with MiDaS estimated depth ranftl2020towardsranftl2021vision. "$\ddagger$" indicates the MiDaS estimated depth from the synthesized image.
• Figure 3: Visualization of the video and depth feature maps and their motion fields without consistency losses. We attribute the inconsistent video-depth output (blue circle) to the inconsistent video-depth feature motions (the last row). This problem exists in multiples layers within the U-Net, and we randomly select two layers for visualization. We follow tumanyan2023plug to visualize the feature maps, and different color in the motion filed indicates different moving direction.
• Figure 4: Comparison between human attribute pre-training (HAP) wang2023disco and our human attribute outpainting pre-training (HAOP). HAP may result in an apparent background mask when the target pose deviates from the original position, while our HAOP mitigates this problem.