Playmate2: Training-Free Multi-Character Audio-Driven Animation via Diffusion Transformer with Reward Feedback

TL;DR

The paper introduces Playmate2, a DiT-based diffusion framework for high-quality audio-driven human videos that supports arbitrarily long sequences and multi-character animation with no training data or architectural changes. It combines a LoRA-based long-video strategy, a partial-update scheme with Direct Preference Optimization to boost lip-sync and natural motion, and a training-free Mask-CFG approach to enable three-or-more character animation. Experimental results on public datasets show superior visual quality, temporal coherence, and multi-character capability compared with state-of-the-art methods, supported by quantitative metrics, qualitative analysis, and user studies. The approach is efficient, scalable, and cost-effective, offering practical benefits for digital humans in entertainment and related domains.

Abstract

Recent advances in diffusion models have significantly improved audio-driven human video generation, surpassing traditional methods in both quality and controllability. However, existing approaches still face challenges in lip-sync accuracy, temporal coherence for long video generation, and multi-character animation. In this work, we propose a diffusion transformer (DiT)-based framework for generating lifelike talking videos of arbitrary length, and introduce a training-free method for multi-character audio-driven animation. First, we employ a LoRA-based training strategy combined with a position shift inference approach, which enables efficient long video generation while preserving the capabilities of the foundation model. Moreover, we combine partial parameter updates with reward feedback to enhance both lip synchronization and natural body motion. Finally, we propose a training-free approach, Mask Classifier-Free Guidance (Mask-CFG), for multi-character animation, which requires no specialized datasets or model modifications and supports audio-driven animation for three or more characters. Experimental results demonstrate that our method outperforms existing state-of-the-art approaches, achieving high-quality, temporally coherent, and multi-character audio-driven video generation in a simple, efficient, and cost-effective manner.

Figures (5)

• Figure 2: Overview of our method. Our framework leverages a LoRA-based training strategy and position shift inference to generate long, temporally coherent videos with consistent identity. A partial parameter update with reward feedback enhances lip synchronization and upper-body motion naturalness. Furthermore, we propose Mask-CFG, a training-free approach for multi-character animation that requires no additional data or model fine-tuning, yet supports audio-driven animation of three or more characters.
• Figure 3: Workflow of Mask-CFG.
• Figure 4: Qualitative comparison with other competing methods.
• Figure 5: Qualitative comparison of long video generation results.
• Figure 6: Visual comparison of DPO ablation study.