AADiff: Audio-Aligned Video Synthesis with Text-to-Image Diffusion

TL;DR

This work tackles the limitation of text-only diffusion-based video synthesis in capturing fine-grained temporal dynamics and audio synchronization. It introduces AADiff, which uses an off-the-shelf text-to-image diffusion model guided by audio embeddings to produce audio-aligned videos, with local editing via attention maps and a sliding-window mechanism to smooth temporal dynamics. The method requires no additional training or paired data and leverages public multimodal models (CLAP, CLIP, Stable Diffusion) to achieve synchronization. Experiments on ESC-50 audio demonstrate alignment between audio intensity and visual changes, demonstrations of animating still images, and insights on the sliding-window trade-off for temporal coherence, highlighting practical utility for content creation.

Abstract

Recent advances in diffusion models have showcased promising results in the text-to-video (T2V) synthesis task. However, as these T2V models solely employ text as the guidance, they tend to struggle in modeling detailed temporal dynamics. In this paper, we introduce a novel T2V framework that additionally employ audio signals to control the temporal dynamics, empowering an off-the-shelf T2I diffusion to generate audio-aligned videos. We propose audio-based regional editing and signal smoothing to strike a good balance between the two contradicting desiderata of video synthesis, i.e., temporal flexibility and coherence. We empirically demonstrate the effectiveness of our method through experiments, and further present practical applications for contents creation.

Figures (9)

• Figure 1: Method overview. Given an audio signal and text prompt, each is first embedded by the audio encoder and the text encoder, respectively. Text tokens with the highest similarities are chosen and used for editing images with hertz2022prompt, where the smoothed audio magnitude controls the attention strength.
• Figure 2: Variable sliding window. A small window size effectively captures dynamic changes, such as thunder, while a larger window size excels at representing gradual transitions, such as wildfire spreading. This hyperparameter allows the content creator to flexibly control the temporal dynamics of the video.
• Figure 3: Qualitative result for different sound sources. Consult the supplementary to check the synchronization between the audio and the output video.
• Figure 4: CLIP similarity and audio magnitude. These two values move in unison, indicating that our model faithfully reflects the audio dynamic in the video semantic.
• Figure 5: Qualitative results with Null-inversion. Our method can be used to combine real images and audio sources to create more immersive audio-visual contents.