AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinement

TL;DR

AnyTalker tackles the data-cost and interaction challenges of multi-person talking video generation by introducing an extensible Audio-Face Cross Attention mechanism built on a diffusion-transformer backbone. It adopts a two-stage training strategy that first learns multi-speaker lip movements from concatenated single-person data and then refines interactivity with a small amount of authentic multi-person data, complemented by a new InteractiveEyes dataset and eye-keypoint motion metric. The approach achieves state-of-the-art lip synchronization, visual quality, and interactivity while substantially reducing data requirements and enabling arbitrary identity scalability. Extensive experiments show that AFCA supports scalable multi-ID driving, and the interactivity metric captures nuanced listener-actor dynamics. This work lays a practical foundation for interactive, multi-person video synthesis in data-constrained settings.

Abstract

Recently, multi-person video generation has started to gain prominence. While a few preliminary works have explored audio-driven multi-person talking video generation, they often face challenges due to the high costs of diverse multi-person data collection and the difficulty of driving multiple identities with coherent interactivity. To address these challenges, we propose AnyTalker, a multi-person generation framework that features an extensible multi-stream processing architecture. Specifically, we extend Diffusion Transformer's attention block with a novel identity-aware attention mechanism that iteratively processes identity-audio pairs, allowing arbitrary scaling of drivable identities. Besides, training multi-person generative models demands massive multi-person data. Our proposed training pipeline depends solely on single-person videos to learn multi-person speaking patterns and refines interactivity with only a few real multi-person clips. Furthermore, we contribute a targeted metric and dataset designed to evaluate the naturalness and interactivity of the generated multi-person videos. Extensive experiments demonstrate that AnyTalker achieves remarkable lip synchronization, visual quality, and natural interactivity, striking a favorable balance between data costs and identity scalability.

Figures (15)

• Figure 1: We propose AnyTalker, a powerful audio-driven framework for interactive multi-person video generation. It can generate natural videos that are rich in gestures, lively emotions, and interactivity, and can freely generalize to arbitrary IDs or even non-human cases.
• Figure 2: (a) The architecture of AnyTalker, which incorporates a novel multi-stream audio processing layer, Audio-Face Cross Attention, enables the handling of multiple facial and audio inputs. (b) The training of AnyTalker is divided into two stages: the first stage uses concatenated multi-person data derived from single-person data mixed with single-person data to learn accurate lip movements; the second stage employs authentic multi-person data to enhance the interactivity in generated videos. (c) The detailed implementation of Audio-Face Cross Attention, a recursively callable structure that applies masking to the output using face masks.
• Figure 3: (a) Mapping of video tokens to audio tokens, facilitated by a custom attention mask. Every 4 audio tokens are bound to 1 video token, except for the first. (b) Mask token used for output masking in Audio-Face Cross Attention.
• Figure 4: Two video clips from InteractiveEyes with ${Motion}$ score (px): left shows original video, right shows cropped face and eye landmarks. Head turn toward the speaker or eyebrow raise will increase $Motion$ and Interactivity; sustained stillness keeps both low.
• Figure 5: Listening and speaking periods of each speaker.