OmniCustom: Sync Audio-Video Customization Via Joint Audio-Video Generation Model

TL;DR

This work defines sync audio-video customization as generating a video that preserves the identity from a reference image while cloning the timbre of a reference audio, with spoken content specified by text. It proposes OmniCustom, a DiT-based framework that injects reference information through separate image and audio LoRA branches into a twin-backbone fusion design, coupled with a contrastive learning objective to reinforce identity and timbre fidelity. The model is trained on OmniCustom-1M, a large-scale audio-visual portrait dataset, and demonstrates state-of-the-art performance in identity preservation, timbre cloning, and audio-visual synchronization, while also supporting background sound generation. The approach achieves strong results in both qualitative and quantitative evaluations and offers practical potential for personalized, synchronized multimedia generation, albeit with current English-language and 5-second-duration constraints that point to future extensions.

Abstract

Existing mainstream video customization methods focus on generating identity-consistent videos based on given reference images and textual prompts. Benefiting from the rapid advancement of joint audio-video generation, this paper proposes a more compelling new task: sync audio-video customization, which aims to synchronously customize both video identity and audio timbre. Specifically, given a reference image $I^{r}$ and a reference audio $A^{r}$, this novel task requires generating videos that maintain the identity of the reference image while imitating the timbre of the reference audio, with spoken content freely specifiable through user-provided textual prompts. To this end, we propose OmniCustom, a powerful DiT-based audio-video customization framework that can synthesize a video following reference image identity, audio timbre, and text prompts all at once in a zero-shot manner. Our framework is built on three key contributions. First, identity and audio timbre control are achieved through separate reference identity and audio LoRA modules that operate through self-attention layers within the base audio-video generation model. Second, we introduce a contrastive learning objective alongside the standard flow matching objective. It uses predicted flows conditioned on reference inputs as positive examples and those without reference conditions as negative examples, thereby enhancing the model ability to preserve identity and timbre. Third, we train OmniCustom on our constructed large-scale, high-quality audio-visual human dataset. Extensive experiments demonstrate that OmniCustom outperforms existing methods in generating audio-video content with consistent identity and timbre fidelity.

Figures (7)

• Figure 1: (a) Overview of our OmniCustom architecture. We extend the joint audio-video generation model OVI low2025ovi by introducing reference image and audio branches alongside the original video and audio flows. The visual and audio VAE encoders project the reference image $I^{r}$ and audio $A^{r}$ into tokens, which are then concatenated with the noised video and audio latent tokens, respectively, before being processed by the fusion blocks. Here, the face embeddings ren2023facial and timbre embeddings ju2024naturalspeech are also input in fusion blocks for further constraint. (b) Fusion Block. It is designed as a symmetric twin backbone with parallel audio and video branches. Our OmniCustom embraces identity and timbre information via finetuning self-attention layers in video and audio branches, respectively. (c) Reference LoRAs. We incorporate separate LoRA into the QKV projections of the reference identity and audio representations. Specifically, the reference identity LoRA is in the self-attention layers of video branch, whereas the reference audio LoRA is in those of audio branch.
• Figure 1: Statistics of our proposed dataset OmniCustom-1M, in terms of age and gender.
• Figure 2: Qualitative comparison with state-of-the-art video customization methods. The speech content of HunyuanCustom hu2025hunyuancustom and Humo chen2025humo is directly determined by their input audio. Our OmniCustom mimics the timbre of the reference audio and flexibly specifies the spoken content through textual prompts.
• Figure 2: Prompt used for audio caption and benchmark construction.
• Figure 3: Ablation study. Face embeddings and the contrastive identity objective boost identity consistency, respectively. See the video results for audio details.