Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GMS-CAVP: Improving Audio-Video Correspondence with Multi-Scale Contrastive and Generative Pretraining

Shentong Mo, Zehua Chen, Jun Zhu

TL;DR

The paper tackles the problem of robust video-audio alignment by moving beyond single-scale, discriminative pretraining. It introduces GMS-CAVP, a framework that jointly learns multi-scale contrastive video-audio alignment and a diffusion-based generative objective conditioned on hierarchical video features. The approach yields a Generative Multi-scale Video-Audio Alignment (MSD) that enables high-fidelity, temporally synchronized audio synthesis, along with a Multi-scale Spatial-temporal Alignment (MSA) for finer cross-modal correspondence. Experiments on VGGSound, AudioSet, and Panda70M show state-of-the-art performance in both video-to-audio generation and retrieval, validating the benefits of combining discriminative and generative objectives across multiple spatial-temporal scales. This work advances cross-modal understanding and creation by enabling more faithful audio generation and more precise audio-visual retrieval, with broad implications for audiovisual AI systems.

Abstract

Recent advances in video-audio (V-A) understanding and generation have increasingly relied on joint V-A embeddings, which serve as the foundation for tasks such as cross-modal retrieval and generation. While prior methods like CAVP effectively model semantic and temporal correspondences between modalities using contrastive objectives, their performance remains suboptimal. A key limitation is the insufficient modeling of the dense, multi-scale nature of both video and audio signals, correspondences often span fine- to coarse-grained spatial-temporal structures, which are underutilized in existing frameworks. To this end, we propose GMS-CAVP, a novel framework that combines Multi-Scale Video-Audio Alignment and Multi-Scale Spatial-Temporal Diffusion-based pretraining objectives to enhance V-A correspondence modeling. First, GMS-CAVP introduces a multi-scale contrastive learning strategy that captures semantic and temporal relations across varying granularities. Second, we go beyond traditional contrastive learning by incorporating a diffusion-based generative objective, enabling modality translation and synthesis between video and audio. This unified discriminative-generative formulation facilitates deeper cross-modal understanding and paves the way for high-fidelity generation. Extensive experiments on VGGSound, AudioSet, and Panda70M demonstrate that GMS-CAVP outperforms previous methods in generation and retrieval.

GMS-CAVP: Improving Audio-Video Correspondence with Multi-Scale Contrastive and Generative Pretraining

TL;DR

The paper tackles the problem of robust video-audio alignment by moving beyond single-scale, discriminative pretraining. It introduces GMS-CAVP, a framework that jointly learns multi-scale contrastive video-audio alignment and a diffusion-based generative objective conditioned on hierarchical video features. The approach yields a Generative Multi-scale Video-Audio Alignment (MSD) that enables high-fidelity, temporally synchronized audio synthesis, along with a Multi-scale Spatial-temporal Alignment (MSA) for finer cross-modal correspondence. Experiments on VGGSound, AudioSet, and Panda70M show state-of-the-art performance in both video-to-audio generation and retrieval, validating the benefits of combining discriminative and generative objectives across multiple spatial-temporal scales. This work advances cross-modal understanding and creation by enabling more faithful audio generation and more precise audio-visual retrieval, with broad implications for audiovisual AI systems.

Abstract

Recent advances in video-audio (V-A) understanding and generation have increasingly relied on joint V-A embeddings, which serve as the foundation for tasks such as cross-modal retrieval and generation. While prior methods like CAVP effectively model semantic and temporal correspondences between modalities using contrastive objectives, their performance remains suboptimal. A key limitation is the insufficient modeling of the dense, multi-scale nature of both video and audio signals, correspondences often span fine- to coarse-grained spatial-temporal structures, which are underutilized in existing frameworks. To this end, we propose GMS-CAVP, a novel framework that combines Multi-Scale Video-Audio Alignment and Multi-Scale Spatial-Temporal Diffusion-based pretraining objectives to enhance V-A correspondence modeling. First, GMS-CAVP introduces a multi-scale contrastive learning strategy that captures semantic and temporal relations across varying granularities. Second, we go beyond traditional contrastive learning by incorporating a diffusion-based generative objective, enabling modality translation and synthesis between video and audio. This unified discriminative-generative formulation facilitates deeper cross-modal understanding and paves the way for high-fidelity generation. Extensive experiments on VGGSound, AudioSet, and Panda70M demonstrate that GMS-CAVP outperforms previous methods in generation and retrieval.
Paper Structure (11 sections, 7 equations, 1 figure, 3 tables)

This paper contains 11 sections, 7 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Preliminaries
  5. Multi-scale Video-Audio Alignment
  6. Generative Multi-scale Video-Audio Alignment
  7. Experiments
  8. Experimental Setup
  9. Comparison to Prior Work
  10. Experimental Analysis
  11. Conclusion

Figures (1)

  • Figure 1: Illustration of the proposed multi-scale discriminative and generative architecture (GMS-CAVP) for learning audio-video correspondence. We introduce the Multi-scale Video-Audio Alignment mechanism, which captures fine-grained hierarchical dependencies for enhanced video-audio correspondence. Then, Generative Multi-scale Video-Audio Alignment is proposed to bridge the generative gap between video and audio representations for improved video-to-audio synthesis.