DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation
Xu Guo, Fulong Ye, Qichao Sun, Liyang Chen, Bingchuan Li, Pengze Zhang, Jiawei Liu, Songtao Zhao, Qian He, Xiangwang Hou
TL;DR
DreamID-Omni introduces a unified framework for controllable human-centric audio-video generation by unifying R2AV, RV2AV, and RA2V within a Symmetric Conditional Diffusion Transformer. It tackles identity-timbre binding and multi-person disentanglement through Dual-Level Disentanglement, combining Syn-RoPE at the signal level with Structured Captions at the semantic level. A three-stage Multi-Task Progressive Training curriculum harmonizes weakly- and strongly-constrained tasks, preventing overfitting and enabling robust cross-task performance. The approach achieves state-of-the-art results across video, audio, and audio-visual consistency on IDBench-Omni, outperforming leading commercial models, and the authors plan to release code to democratize access to this capability.
Abstract
Recent advancements in foundation models have revolutionized joint audio-video generation. However, existing approaches typically treat human-centric tasks including reference-based audio-video generation (R2AV), video editing (RV2AV) and audio-driven video animation (RA2V) as isolated objectives. Furthermore, achieving precise, disentangled control over multiple character identities and voice timbres within a single framework remains an open challenge. In this paper, we propose DreamID-Omni, a unified framework for controllable human-centric audio-video generation. Specifically, we design a Symmetric Conditional Diffusion Transformer that integrates heterogeneous conditioning signals via a symmetric conditional injection scheme. To resolve the pervasive identity-timbre binding failures and speaker confusion in multi-person scenarios, we introduce a Dual-Level Disentanglement strategy: Synchronized RoPE at the signal level to ensure rigid attention-space binding, and Structured Captions at the semantic level to establish explicit attribute-subject mappings. Furthermore, we devise a Multi-Task Progressive Training scheme that leverages weakly-constrained generative priors to regularize strongly-constrained tasks, preventing overfitting and harmonizing disparate objectives. Extensive experiments demonstrate that DreamID-Omni achieves comprehensive state-of-the-art performance across video, audio, and audio-visual consistency, even outperforming leading proprietary commercial models. We will release our code to bridge the gap between academic research and commercial-grade applications.