Dancing Avatar: Pose and Text-Guided Human Motion Videos Synthesis with Image Diffusion Model

TL;DR

Dancing Avatar tackles the problem of generating pose- and text-guided human motion videos by repurposing a pretrained text-to-image diffusion model in an autoregressive frame sequence. It introduces three alignment modules—intra-frame for consistent character appearance, background alignment for stable backdrops, and inter-frame for temporal coherence—alongside ChatGPT-informed prompt refinement and segment-anything-based background processing. Through extensive qualitative and quantitative evaluations, the method demonstrates superior frame quality, pose accuracy, background fidelity, and temporal stability compared with state-of-the-art pose/text-guided video synthesis methods. This approach enables high-fidelity avatar video synthesis without fine-tuning to a dedicated T2V model, with broad implications for controllable digital avatars and animation from textual and pose inputs.

Abstract

The rising demand for creating lifelike avatars in the digital realm has led to an increased need for generating high-quality human videos guided by textual descriptions and poses. We propose Dancing Avatar, designed to fabricate human motion videos driven by poses and textual cues. Our approach employs a pretrained T2I diffusion model to generate each video frame in an autoregressive fashion. The crux of innovation lies in our adept utilization of the T2I diffusion model for producing video frames successively while preserving contextual relevance. We surmount the hurdles posed by maintaining human character and clothing consistency across varying poses, along with upholding the background's continuity amidst diverse human movements. To ensure consistent human appearances across the entire video, we devise an intra-frame alignment module. This module assimilates text-guided synthesized human character knowledge into the pretrained T2I diffusion model, synergizing insights from ChatGPT. For preserving background continuity, we put forth a background alignment pipeline, amalgamating insights from segment anything and image inpainting techniques. Furthermore, we propose an inter-frame alignment module that draws inspiration from an auto-regressive pipeline to augment temporal consistency between adjacent frames, where the preceding frame guides the synthesis process of the current frame. Comparisons with state-of-the-art methods demonstrate that Dancing Avatar exhibits the capacity to generate human videos with markedly superior quality, both in terms of human and background fidelity, as well as temporal coherence compared to existing state-of-the-art approaches.

Figures (3)

• Figure 1: Pipeline of Dancing Avatar. The Dancing Avatar pipeline begins by acquiring the intra-frame alignment module. It then executes the background alignment process. Ultimately, it synthesizes each frame of the human motion video in an auto-regressive fashion, assisted by the inter-frame alignment module.
• Figure 2: Comparison between Dancing Avatar and previous state-of-the-art pose and text-guided human motion video synthesis approaches. The videos synthesized by Dancing Avatar exhibit higher video quality, alignment with inputs, and temporal coherence maintained across successive frames.
• Figure 3: Visualized result of ablation experiment. Dancing Avatar demonstrates notably greater consistency between adjacent frames in comparison to its ablation counterparts.