TCAN: Animating Human Images with Temporally Consistent Pose Guidance using Diffusion Models

TL;DR

TCAN tackles temporally consistent, pose-driven human image animation under noisy pose estimation. It leverages a frozen pre-trained ControlNet with an Appearance-Pose Adaptation (APPA) layer, aided by a Temporal ControlNet and a Pose-driven Temperature Map to disentangle appearance from pose and stabilize the background. The method follows a two-stage training procedure, with long-term video generation enabled by MultiDiffusion, and demonstrates strong generalization on the TikTok dataset and to animation characters via pose retargeting. Results indicate state-of-the-art performance across image- and video-level metrics, with qualitative and user-study evidence supporting improved temporal coherence and identity preservation, while acknowledging potential misuse and the need for verification methods.

Abstract

Pose-driven human-image animation diffusion models have shown remarkable capabilities in realistic human video synthesis. Despite the promising results achieved by previous approaches, challenges persist in achieving temporally consistent animation and ensuring robustness with off-the-shelf pose detectors. In this paper, we present TCAN, a pose-driven human image animation method that is robust to erroneous poses and consistent over time. In contrast to previous methods, we utilize the pre-trained ControlNet without fine-tuning to leverage its extensive pre-acquired knowledge from numerous pose-image-caption pairs. To keep the ControlNet frozen, we adapt LoRA to the UNet layers, enabling the network to align the latent space between the pose and appearance features. Additionally, by introducing an additional temporal layer to the ControlNet, we enhance robustness against outliers of the pose detector. Through the analysis of attention maps over the temporal axis, we also designed a novel temperature map leveraging pose information, allowing for a more static background. Extensive experiments demonstrate that the proposed method can achieve promising results in video synthesis tasks encompassing various poses, like chibi. Project Page: https://eccv2024tcan.github.io/

Figures (12)

• Figure 1: Generated results of TCAN: The first two rows show the results on the TikTok dataset, and the last two rows show the results on chibi animation characters. All results are generated using TCAN trained on the TikTok dataset. Severely erroneous input poses are highlighted in red. Note that the proposed TCAN can generalize to poses with outliers and unusual ratios, such as those of chibi characters.
• Figure 2: Our method involves a two-stage training strategy. We randomly select two images from the training video and use them as the source image and the driving image, respectively. In the first stage, the appearance UNet and the APPA layer are trained conditioning on the source image while the ControlNet is frozen. In the second stage, we train the temporal layers in the denoising UNet and ControlNet.
• Figure 3: Ablation on the APPA layer. Using only the frozen ControlNet significantly deteriorates texture quality, while the APPA layer accurately preserves the style of both the foreground and background from the source image in the output, showing notable differences.
• Figure 4: Visualization of the attention maps from the temporal transformer block at a resolution of $64\times64$. Best viewed when zoomed in.
• Figure 5: Qualitative comparison with baselines on the TikTok jafarian2021learning dataset. Frames from segments of the driving video are arranged sequentially.