SyncFlow: Toward Temporally Aligned Joint Audio-Video Generation from Text

TL;DR

This work tackles the challenge of generating temporally synchronized audio and video from text. It introduces SyncFlow, a dual-diffusion-transformer (d-DiT) framework built on latent rectifier flow matching and a modality-adapted, multi-stage training strategy to jointly model video and audio. The approach yields higher audio-visual correlation and audio quality than cascaded or contrastive-encoder baselines and demonstrates strong zero-shot capabilities, including video resolution adaptation. These results suggest a practical path toward unified, text-conditioned audio-video generation with improved synchronization and efficiency.

Abstract

Video and audio are closely correlated modalities that humans naturally perceive together. While recent advancements have enabled the generation of audio or video from text, producing both modalities simultaneously still typically relies on either a cascaded process or multi-modal contrastive encoders. These approaches, however, often lead to suboptimal results due to inherent information losses during inference and conditioning. In this paper, we introduce SyncFlow, a system that is capable of simultaneously generating temporally synchronized audio and video from text. The core of SyncFlow is the proposed dual-diffusion-transformer (d-DiT) architecture, which enables joint video and audio modelling with proper information fusion. To efficiently manage the computational cost of joint audio and video modelling, SyncFlow utilizes a multi-stage training strategy that separates video and audio learning before joint fine-tuning. Our empirical evaluations demonstrate that SyncFlow produces audio and video outputs that are more correlated than baseline methods with significantly enhanced audio quality and audio-visual correspondence. Moreover, we demonstrate strong zero-shot capabilities of SyncFlow, including zero-shot video-to-audio generation and adaptation to novel video resolutions without further training.

Figures (6)

• Figure 1: The main architecture of dual-diffusion-transformer (d-DiT) used by SyncFlow. Two parallel towers handle video and audio generation, with modality adaptors to enhance synchronization. Text input conditions the video generation towers through cross-attentions.
• Figure 2: The detailed implementation of the spatial-temporal attention layers, audio transformer layers, and modality adaptor. The output of the modality adaptor is concatenated with the flow matching time step embedding as the cross-attention condition to the audio transformer layer.
• Figure 3: Snapshot of video and audio generated by SyncFlow. The video frames are displayed every four frames for simplicity. The original audio frame length corresponding to each audio is $32$.
• Figure 4: Example of zero-shot video-to-audio generation using SyncFlow. The input video is sourced from the VGGSound evaluation set.
• Figure 5: The effect of classifier-free guidance scale on the performance of SyncFlow-VGG.