Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Simple but Strong Baseline for Sounding Video Generation: Effective Adaptation of Audio and Video Diffusion Models for Joint Generation

Masato Ishii, Akio Hayakawa, Takashi Shibuya, Yuki Mitsufuji

TL;DR

This work presents a simple, strong baseline for sounding video generation by reusing pre-trained audio and video diffusion models and training only lightweight added modules to enable joint generation. It introduces two mechanisms—timestep adjustment and Cross-Modal Conditioning as Positional Encoding (CMC-PE)—to improve temporal alignment between generated audio and video. Through experiments on GreatestHits and benchmark datasets (Landscape, VGGSound), the approach achieves improved AV alignment and competitive audio/video quality while substantially reducing training cost compared to fully joint models. The findings suggest that careful temporal alignment biases and efficient cross-modal conditioning can yield high-quality, synchronized audio-video generation with minimal architectural changes.

Abstract

In this work, we build a simple but strong baseline for sounding video generation. Given base diffusion models for audio and video, we integrate them with additional modules into a single model and train it to make the model jointly generate audio and video. To enhance alignment between audio-video pairs, we introduce two novel mechanisms in our model. The first one is timestep adjustment, which provides different timestep information to each base model. It is designed to align how samples are generated along with timesteps across modalities. The second one is a new design of the additional modules, termed Cross-Modal Conditioning as Positional Encoding (CMC-PE). In CMC-PE, cross-modal information is embedded as if it represents temporal position information, and the embeddings are fed into the model like positional encoding. Compared with the popular cross-attention mechanism, CMC-PE provides a better inductive bias for temporal alignment in the generated data. Experimental results validate the effectiveness of the two newly introduced mechanisms and also demonstrate that our method outperforms existing methods.

A Simple but Strong Baseline for Sounding Video Generation: Effective Adaptation of Audio and Video Diffusion Models for Joint Generation

TL;DR

This work presents a simple, strong baseline for sounding video generation by reusing pre-trained audio and video diffusion models and training only lightweight added modules to enable joint generation. It introduces two mechanisms—timestep adjustment and Cross-Modal Conditioning as Positional Encoding (CMC-PE)—to improve temporal alignment between generated audio and video. Through experiments on GreatestHits and benchmark datasets (Landscape, VGGSound), the approach achieves improved AV alignment and competitive audio/video quality while substantially reducing training cost compared to fully joint models. The findings suggest that careful temporal alignment biases and efficient cross-modal conditioning can yield high-quality, synchronized audio-video generation with minimal architectural changes.

Abstract

In this work, we build a simple but strong baseline for sounding video generation. Given base diffusion models for audio and video, we integrate them with additional modules into a single model and train it to make the model jointly generate audio and video. To enhance alignment between audio-video pairs, we introduce two novel mechanisms in our model. The first one is timestep adjustment, which provides different timestep information to each base model. It is designed to align how samples are generated along with timesteps across modalities. The second one is a new design of the additional modules, termed Cross-Modal Conditioning as Positional Encoding (CMC-PE). In CMC-PE, cross-modal information is embedded as if it represents temporal position information, and the embeddings are fed into the model like positional encoding. Compared with the popular cross-attention mechanism, CMC-PE provides a better inductive bias for temporal alignment in the generated data. Experimental results validate the effectiveness of the two newly introduced mechanisms and also demonstrate that our method outperforms existing methods.
Paper Structure (34 sections, 11 equations, 8 figures, 3 tables, 1 algorithm)

This paper contains 34 sections, 11 equations, 8 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and related work
  3. Diffusion models
  4. Basics
  5. Application to single-modal generation
  6. Audio-video generative models
  7. Cross-modal conditional generation
  8. Audio-video joint generation
  9. Proposed method
  10. Overview
  11. Timestep adjustment
  12. Why do we need to adjust timesteps across modalities?
  13. A simple solution for timestep adjustment
  14. How to feed cross-modal features into U-Net
  15. The standard choice: cross-attention and its limitation
  16. ...and 19 more sections

Figures (8)

  • Figure 1: Overview of proposed model. For brevity of the diagram, we omit encoders to obtain latent features and paths for textual conditioning from both base models.
  • Figure 2: w/o timestep adjustment
  • Figure 3: w/ timestep adjustment
  • Figure 5: Cross-attention
  • Figure 6: CMC-PE
  • ...and 3 more figures