Free-viewpoint Human Animation with Pose-correlated Reference Selection

TL;DR

The paper tackles free-viewpoint human animation under dramatic viewpoint and camera-distance changes, where single-reference approaches struggle to preserve appearance. It introduces a diffusion-based framework with a pose-correlated reference selection mechanism that leverages multiple reference images, a pose correlation module to generate informative correlation maps, and an adaptive reference selection strategy to fuse the most relevant features efficiently. The method is trained on a new Multi-Shot TED Video Dataset (MSTed) and the public DyMVHumans dataset, and it achieves state-of-the-art results under large viewpoint changes, even when only a single reference image is available at inference due to training with multiple references. Additionally, the work provides ablation evidence for the effectiveness of the reference multiplicity and correlation-guided feature selection, demonstrating improved appearance consistency and video quality for free-viewpoint synthesis with diffusion models.

Abstract

Diffusion-based human animation aims to animate a human character based on a source human image as well as driving signals such as a sequence of poses. Leveraging the generative capacity of diffusion model, existing approaches are able to generate high-fidelity poses, but struggle with significant viewpoint changes, especially in zoom-in/zoom-out scenarios where camera-character distance varies. This limits the applications such as cinematic shot type plan or camera control. We propose a pose-correlated reference selection diffusion network, supporting substantial viewpoint variations in human animation. Our key idea is to enable the network to utilize multiple reference images as input, since significant viewpoint changes often lead to missing appearance details on the human body. To eliminate the computational cost, we first introduce a novel pose correlation module to compute similarities between non-aligned target and source poses, and then propose an adaptive reference selection strategy, utilizing the attention map to identify key regions for animation generation. To train our model, we curated a large dataset from public TED talks featuring varied shots of the same character, helping the model learn synthesis for different perspectives. Our experimental results show that with the same number of reference images, our model performs favorably compared to the current SOTA methods under large viewpoint change. We further show that the adaptive reference selection is able to choose the most relevant reference regions to generate humans under free viewpoints.

Figures (6)

• Figure 1: In this work, we aim to address the challenging task of free-viewpoint human animation synthesis under large viewpoint and camera distance changes. Our proposed method successfully generates novel-view videos with consistent character appearance across substantial viewpoint shifts.
• Figure 2: The illustration of our framework. Our framework feed a reference set $\{\mathbf{I}_{ref}^i\}_{i=1}^N$ into reference Unet ($\mathcal{R}$) to extract the reference feature. To filter out the redundant information in reference features set, we propose a pose correlation guider to create a correlation map to indicate the informative region of the reference spatially. Moreover, we adopt a reference selection strategy to pick up the informative tokens from the reference feature set according to the correlation map and pass them to the following modules.
• Figure 3: The illustration of pose correlation module (PCM). Each reference pose $\mathbf{P}_{ref}^i$ will be fed into PCM with each target pose $\mathbf{P}_{tgt}^j$ to compute a correlation map, which indicate the informative region of input reference image.
• Figure 4: Qualitative results in Multi-Shot Ted dataset and DyMVHuman dataset. Compared with compared methods, our model can achieve high quality while maintaining the appearance consistency.
• Figure 5: The visualization of our ablation study. $\mathcal{H}$ is the pose correlation module. We can observe that with mulitple reference and pose correlation module, our method can obtain the best results.