StableAnimator: High-Quality Identity-Preserving Human Image Animation

TL;DR

StableAnimator tackles identity drift in pose-driven human image animation by introducing an end-to-end ID-preserving video diffusion framework. It couples a Global Content-aware Face Encoder with a Distribution-aware ID Adapter to fuse face and image embeddings into a video diffusion backbone while mitigating temporal distortion. At inference, an HJB-based face optimization guides the denoising process to further enhance face quality without external post-processing. Across TikTok and Unseen100 benchmarks, the approach achieves superior ID preservation and video fidelity, supported by ablations and user studies, signaling a meaningful advance in identity-consistent video animation.

Abstract

Current diffusion models for human image animation struggle to ensure identity (ID) consistency. This paper presents StableAnimator, the first end-to-end ID-preserving video diffusion framework, which synthesizes high-quality videos without any post-processing, conditioned on a reference image and a sequence of poses. Building upon a video diffusion model, StableAnimator contains carefully designed modules for both training and inference striving for identity consistency. In particular, StableAnimator begins by computing image and face embeddings with off-the-shelf extractors, respectively and face embeddings are further refined by interacting with image embeddings using a global content-aware Face Encoder. Then, StableAnimator introduces a novel distribution-aware ID Adapter that prevents interference caused by temporal layers while preserving ID via alignment. During inference, we propose a novel Hamilton-Jacobi-Bellman (HJB) equation-based optimization to further enhance the face quality. We demonstrate that solving the HJB equation can be integrated into the diffusion denoising process, and the resulting solution constrains the denoising path and thus benefits ID preservation. Experiments on multiple benchmarks show the effectiveness of StableAnimator both qualitatively and quantitatively.

Figures (20)

• Figure 1: Pose-driven Human image animations generated by StableAnimator, showing its power to synthesize high-fidelity and ID-preserving videos. FaceFusion facefusion is a face-swapping tool. GFP-GAN wang2021gfpgan and CodeFormer zhou2022codeformer are face restoration models. ControlNeXt peng2024controlnext is the latest open-source animation model.
• Figure 2: Architecture of StableAnimator. (a) and (b) refer to the structure of the Face Encoder and each block in the U-Net. Embeddings from the Image Encoder and Face Encoder are injected to each block of U-Net. Given the reference, we extract the image embeddings and face embeddings utilizing Image Encoder and Arcface. The face embeddings are fed into the FaceEncoder to enhance ID. Then, image embeddings and refined face embeddings are injected into the U-Net through the ID Adapter to ensure ID consistency.
• Figure 3: Qualitative comparisons with state-of-the-art methods. More examples can be found in the supplementary material.
• Figure 4: Ablations on core components of StableAnimator.
• Figure 5: Ablation study on face enhancement strategies.