DisPose: Disentangling Pose Guidance for Controllable Human Image Animation

TL;DR

DisPose tackles controllable human image animation by disentangling pose guidance into motion field guidance and keypoint correspondence derived from a skeleton pose and a reference image. It introduces a reference-based dense motion field via conditional motion propagation and diffusion-feature keypoint embeddings, integrated through a plug-and-play Hybrid ControlNet that freezes the baseline model. The approach yields improved video fidelity, temporal consistency, and cross-identity robustness, demonstrated across MusePose and MimicMotion with evaluations on TikTok and unseen data. The method offers a practical, generalizable path to richer control signals without requiring dense inputs, facilitating broader applicability in animating diverse identities.

Abstract

Controllable human image animation aims to generate videos from reference images using driving videos. Due to the limited control signals provided by sparse guidance (e.g., skeleton pose), recent works have attempted to introduce additional dense conditions (e.g., depth map) to ensure motion alignment. However, such strict dense guidance impairs the quality of the generated video when the body shape of the reference character differs significantly from that of the driving video. In this paper, we present DisPose to mine more generalizable and effective control signals without additional dense input, which disentangles the sparse skeleton pose in human image animation into motion field guidance and keypoint correspondence. Specifically, we generate a dense motion field from a sparse motion field and the reference image, which provides region-level dense guidance while maintaining the generalization of the sparse pose control. We also extract diffusion features corresponding to pose keypoints from the reference image, and then these point features are transferred to the target pose to provide distinct identity information. To seamlessly integrate into existing models, we propose a plug-and-play hybrid ControlNet that improves the quality and consistency of generated videos while freezing the existing model parameters. Extensive qualitative and quantitative experiments demonstrate the superiority of DisPose compared to current methods. Project page: \href{https://github.com/lihxxx/DisPose}{https://github.com/lihxxx/DisPose}.

Figures (15)

• Figure 1: Our method demonstrates its ability to produce diverse animations and preserve consistency of appearance.
• Figure 2: The overview of proposed DisPose.
• Figure 3: Qualitative comparisons between our method and the state-of-the-art models on the TikTok dataset.
• Figure 4: Qualitative comparison of our approach with the dense control-based method.
• Figure 5: Comparison of our reference-based dense motion field and existing dense conditions.