PoseCrafter: One-Shot Personalized Video Synthesis Following Flexible Pose Control

TL;DR

PoseCrafter addresses the challenge of one-shot personalized video synthesis guided by flexible pose controls without requiring ground-truth target frames. It combines a Stable Diffusion backbone with ControlNet and introduces reference-frame inversion, reference-frame insertion, and latent editing via affine transformations to preserve identity and refine facial/hand details. Two temporal modules—Key-Frame Attention and Temporal Attention—enable cross-frame consistency and temporal coherence during generation. Evaluations on TikTok, TED, and open-domain videos show PoseCrafter often outperforms baselines trained on large video datasets, delivering high fidelity, faithful pose alignment, and robust identity preservation with efficient one-shot fine-tuning. The approach enables flexible pose transfer from the same or different individuals and supports artificial pose design, highlighting practical potential for lightweight, identity-preserving pose-guided video synthesis, while noting limitations with highly complex poses and notable ethical considerations.

Abstract

In this paper, we introduce PoseCrafter, a one-shot method for personalized video generation following the control of flexible poses. Built upon Stable Diffusion and ControlNet, we carefully design an inference process to produce high-quality videos without the corresponding ground-truth frames. First, we select an appropriate reference frame from the training video and invert it to initialize all latent variables for generation. Then, we insert the corresponding training pose into the target pose sequences to enhance faithfulness through a trained temporal attention module. Furthermore, to alleviate the face and hand degradation resulting from discrepancies between poses of training videos and inference poses, we implement simple latent editing through an affine transformation matrix involving facial and hand landmarks. Extensive experiments on several datasets demonstrate that PoseCrafter achieves superior results to baselines pre-trained on a vast collection of videos under 8 commonly used metrics. Besides, PoseCrafter can follow poses from different individuals or artificial edits and simultaneously retain the human identity in an open-domain training video. Our project page is available at https://ml-gsai.github.io/PoseCrafter-demo/.

Figures (11)

• Figure 1: Inference framework of PoseCrafter. First, we select a frame from the training video to form a pseudo reference video, followed by DDIM inversion. Next, the pose from the reference frame is inserted into the inference poses. Finally, we edit latent to refine the generation of faces and hands by affine transformation.
• Figure 2: Qualitative Comparisons of all methods ($M=100$). Left: TED. Right: TikTok. Time progresses from left to right. PoseCrafter yields the highest quality videos.
• Figure 3: Inference from poses of the same individual ($N=100$ and $M=100$). Time progresses from left to right.
• Figure 4: Inference from artificially designed poses ($N=100$ and $M=8,8,16$). Time progresses from left to right. From top to bottom: the first pose sequence features a right eye blink; the second pose sequence involves tilting the head rightward; the third pose sequence is waving an arm followed by shaking the head.
• Figure 5: Inference poses of other individuals ($N=50$ and $M=50$). Time progresses from left to right.