DeCo: Decoupled Human-Centered Diffusion Video Editing with Motion Consistency

TL;DR

DeCo tackles the challenge of editing human-centered videos with diffusion models by decoupling foreground humans and background content into separate editable components. It introduces a decoupled dynamic human representation based on the parametric body prior $SMPL-X$ and a background atlas edited via depth-guided diffusion, together with a SDS-based optimization extended into $z^n$ and $z^r$ spaces to improve geometry and texture, and a lighting-aware video harmonizer to maintain illumination consistency. The method demonstrates improved frame-level consistency and textual faithfulness on several datasets, outperforming baselines like Tune-A-Video, StableVideo, TokenFlow, and Rerender, especially in longer videos. The work offers a practical pipeline for realistic, text-guided edits of humans in video with controllable appearance and motion while preserving original dynamics.

Abstract

Diffusion models usher a new era of video editing, flexibly manipulating the video contents with text prompts. Despite the widespread application demand in editing human-centered videos, these models face significant challenges in handling complex objects like humans. In this paper, we introduce DeCo, a novel video editing framework specifically designed to treat humans and the background as separate editable targets, ensuring global spatial-temporal consistency by maintaining the coherence of each individual component. Specifically, we propose a decoupled dynamic human representation that utilizes a parametric human body prior to generate tailored humans while preserving the consistent motions as the original video. In addition, we consider the background as a layered atlas to apply text-guided image editing approaches on it. To further enhance the geometry and texture of humans during the optimization, we extend the calculation of score distillation sampling into normal space and image space. Moreover, we tackle inconsistent lighting between the edited targets by leveraging a lighting-aware video harmonizer, a problem previously overlooked in decompose-edit-combine approaches. Extensive qualitative and numerical experiments demonstrate that DeCo outperforms prior video editing methods in human-centered videos, especially in longer videos.

Figures (6)

• Figure 1: Editing results. Given text prompts, our proposed DeCo enables independent and combined edits of both background and humans while preserving the original structure and motions. Best viewed in zoom.
• Figure 2: Framework of DeCo. The framework is divided into three parts. 1) We utilize NLA kasten2021layered to create a background atlas for the input video, with the depth-guided diffusion model $D_b$ generating scenes based on background prompts. 2) The diffusion model $D_h$ optimizes shape parameters $\beta$, expressive parameters $\psi$, displacement layer $\mathbf{D}$, and texture map $\mathbf{\Psi}$ of $\hat{\mathbf{T}}$ using $\mathcal{L}_{\mathrm{geo+tex}}$ and $\mathcal{L}_{\mathrm{norm+rgb}}$. The RGB image $I_r$ and normal map $I^N$ rendered from $\hat{\mathbf{T}}$ are encoded into latent vectors $z^r$ and $z^n$ for $\mathcal{L}_{\mathrm{geo+tex}}$, then denoised to $z^r_0$ and $z^n_0$ for $\mathcal{L}_{\mathrm{norm+rgb}}$. $\hat{\mathbf{T}}$ is animated using pose parameters estimated from the original video. 3) We composite the edited targets with a video harmonizer to ensure the harmonization between them.
• Figure 3: Qualitative comparisons of our DeCo and four representative video editing methods. DeCo achieves much more satisfactory results in terms of video quality and motion consistency.
• Figure 4: Text-guided human generation methods. TADA liao2023tada encounters several issues, such as cartoon-like body shapes ($\textit{Sherlock Holmes}$), blurry patterns ($\textit{Spiderman}$), and misalignment with the prompts ($\textit{Aquaman}$).
• Figure 5: Effects of the loss functions. The two prompts used from top to bottom are "Barack Obama" and "Joker". Unnatural head-to-shoulder ratio, blurred textures, and distorted body shapes are shown in red boxes.