JoyAvatar: Unlocking Highly Expressive Avatars via Harmonized Text-Audio Conditioning

TL;DR

JoyAvatar tackles the challenge of aligning avatar motion to complex text prompts in long-duration videos by introducing a Twin-Teacher Enhanced DMD post-training framework and a dynamic CFG strategy. The method decouples multimodal conditioning—audio, text, and reference imagery—through a dual-teacher objective and timestep-aware guidance, enabling natural full-body motion, dynamic camera trajectories, and reliable lip-sync. It leverages FramePack-encoded motion and a pseudo last frame to maintain identity while enabling unlimited duration. Experiments show state-of-the-art performance against Omnihuman-1.5 and KlingAvatar 2.0, and the approach supports multi-person dialogues and non-human subjects without extra data. This improves text controllability and versatility for cinematic and interactive avatar applications.

Abstract

Existing video avatar models have demonstrated impressive capabilities in scenarios such as talking, public speaking, and singing. However, the majority of these methods exhibit limited alignment with respect to text instructions, particularly when the prompts involve complex elements including large full-body movement, dynamic camera trajectory, background transitions, or human-object interactions. To break out this limitation, we present JoyAvatar, a framework capable of generating long duration avatar videos, featuring two key technical innovations. Firstly, we introduce a twin-teacher enhanced training algorithm that enables the model to transfer inherent text-controllability from the foundation model while simultaneously learning audio-visual synchronization. Secondly, during training, we dynamically modulate the strength of multi-modal conditions (e.g., audio and text) based on the distinct denoising timestep, aiming to mitigate conflicts between the heterogeneous conditioning signals. These two key designs serve to substantially expand the avatar model's capacity to generate natural, temporally coherent full-body motions and dynamic camera movements as well as preserve the basic avatar capabilities, such as accurate lip-sync and identity consistency. GSB evaluation results demonstrate that our JoyAvatar model outperforms the state-of-the-art models such as Omnihuman-1.5 and KlingAvatar 2.0. Moreover, our approach enables complex applications including multi-person dialogues and non-human subjects role-playing. Some video samples are provided on https://joyavatar.github.io/.

Figures (10)

• Figure 1: JoyAvatar model can generate vivid avatar videos with the input prompts involving complex motion patterns (The reference images are marked with orange borders, and the text prompt is displayed below the video).
• Figure 2: Visualization of GSB results comparing our method with OmniHuman-1.5 and KlingAvatar 2.0 across various evaluation dimensions.
• Figure 3: Some notable phenomena in existing avatar models: insufficient control over text prompts (Key actions in the prompts are highlighted in green and generated frames that faithfully adhere to the input prompt are marked with green borders).
• Figure 4: Overview of our JoyAvatar pretraining pipeline (Yellow-bordered frames denote noise-free frames while gray-bordered frames indicate noisy frames).
• Figure 5: Pipeline of Twin-Teacher Enhanced DMD Post-Training.