MM-Sonate: Multimodal Controllable Audio-Video Generation with Zero-Shot Voice Cloning

TL;DR

MM-Sonate tackles the challenge of synchronized audio-video generation with fine-grained acoustic control by introducing a unified instruction-phoneme conditioning and a timbre injection mechanism that enables zero-shot voice cloning within joint AV synthesis. It leverages a multimodal diffusion transformer (MM-DiT) with flow matching, supported by a large-scale multimodal pretraining dataset and a high-fidelity synthetic timbre dataset, to achieve state-of-the-art lip synchronization, speech intelligibility, and cloning fidelity. A noise-based negative conditioning strategy for classifier-free guidance further boosts acoustic fidelity, while stochastic modality masking enables flexible multi-task training across T2VA, TI2VA, TA2VA, and TIA2VA tasks. The approach yields strong practical impact for dubbing, animation, and multimedia content creation, with robust safety and ethics safeguards such as watermarks and controlled access.

Abstract

Joint audio-video generation aims to synthesize synchronized multisensory content, yet current unified models struggle with fine-grained acoustic control, particularly for identity-preserving speech. Existing approaches either suffer from temporal misalignment due to cascaded generation or lack the capability to perform zero-shot voice cloning within a joint synthesis framework. In this work, we present MM-Sonate, a multimodal flow-matching framework that unifies controllable audio-video joint generation with zero-shot voice cloning capabilities. Unlike prior works that rely on coarse semantic descriptions, MM-Sonate utilizes a unified instruction-phoneme input to enforce strict linguistic and temporal alignment. To enable zero-shot voice cloning, we introduce a timbre injection mechanism that effectively decouples speaker identity from linguistic content. Furthermore, addressing the limitations of standard classifier-free guidance in multimodal settings, we propose a noise-based negative conditioning strategy that utilizes natural noise priors to significantly enhance acoustic fidelity. Empirical evaluations demonstrate that MM-Sonate establishes new state-of-the-art performance in joint generation benchmarks, significantly outperforming baselines in lip synchronization and speech intelligibility, while achieving voice cloning fidelity comparable to specialized Text-to-Speech systems.

Figures (7)

• Figure 1: Audio–Video Joint Generation with Multimodal Control: synthesizing coherent, temporally aligned dialogue conditioned on the first-frame and distinct reference audios. Unlike existing frameworks, our unified architecture supports flexible combinations of multi-modal inputs for precise control. MM-Sonate outperforms baselines (Ovi, CosyVoice2) on 9/11 metrics.
• Figure 2: The framework is a multimodal flow-matching model enabling joint audio-video generation with fine-grained control. The unified instructions combining video/audio captions with phoneme sequences for precise content alignment. The reference audio for zero-shot voice cloning. The first frame image for visual conditioning. A dedicated mechanism injects speaker embeddings (from reference audio) and ID embeddings (for multi-speaker control) directly into the phoneme sequence via element-wise addition. These multimodal features are then fused and processed by the MM-DiT backbone to model the joint distribution of audio and video latents.
• Figure 3: The data construction pipeline for the high-fidelity synthetic timbre dataset. The process begins with extracting clean acoustic prompts from raw speech and music via denoising and vocal separation tools. These prompts condition DMP-TTS yin2025dmp to synthesize speech from neutral texts. Finally, a WavLM-basedchen2022wavlm speaker verification module filters out samples with low cosine similarity to ensure high speaker identity preservation.
• Figure 4: Performance comparison of different negative conditioning strategies.
• Figure 5: Sensitivity analysis of negative embedding energy on generation metrics.