Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Masked Generative Video-to-Audio Transformers with Enhanced Synchronicity

Santiago Pascual, Chunghsin Yeh, Ioannis Tsiamas, Joan Serrà

TL;DR

MaskVAT addresses the challenge of generating synchronized, high-quality audio from silent video by integrating a full-band neural audio codec with a masked sequence-to-sequence transformer and multi-modal visual conditioning. The model leverages a DAC-based codegram tokenizer, three conditioning architectures (AdaLN, Seq2Seq, Hybrid), and a post-sampling beam selection to optimize alignment with visual events. It introduces a cosine masking schedule for training, regression and contrastive losses with BEATs for semantic consistency, and a SCAV-based beam selection to pick the best alignment with the input video. Empirically, MaskVAT achieves superior synchronization and competitive audio quality on VGGSound and MUSIC across objective and subjective evaluations, demonstrating the practical potential for synchronized V2A in media production and cross-modal synthesis.

Abstract

Video-to-audio (V2A) generation leverages visual-only video features to render plausible sounds that match the scene. Importantly, the generated sound onsets should match the visual actions that are aligned with them, otherwise unnatural synchronization artifacts arise. Recent works have explored the progression of conditioning sound generators on still images and then video features, focusing on quality and semantic matching while ignoring synchronization, or by sacrificing some amount of quality to focus on improving synchronization only. In this work, we propose a V2A generative model, named MaskVAT, that interconnects a full-band high-quality general audio codec with a sequence-to-sequence masked generative model. This combination allows modeling both high audio quality, semantic matching, and temporal synchronicity at the same time. Our results show that, by combining a high-quality codec with the proper pre-trained audio-visual features and a sequence-to-sequence parallel structure, we are able to yield highly synchronized results on one hand, whilst being competitive with the state of the art of non-codec generative audio models. Sample videos and generated audios are available at https://maskvat.github.io .

Masked Generative Video-to-Audio Transformers with Enhanced Synchronicity

TL;DR

MaskVAT addresses the challenge of generating synchronized, high-quality audio from silent video by integrating a full-band neural audio codec with a masked sequence-to-sequence transformer and multi-modal visual conditioning. The model leverages a DAC-based codegram tokenizer, three conditioning architectures (AdaLN, Seq2Seq, Hybrid), and a post-sampling beam selection to optimize alignment with visual events. It introduces a cosine masking schedule for training, regression and contrastive losses with BEATs for semantic consistency, and a SCAV-based beam selection to pick the best alignment with the input video. Empirically, MaskVAT achieves superior synchronization and competitive audio quality on VGGSound and MUSIC across objective and subjective evaluations, demonstrating the practical potential for synchronized V2A in media production and cross-modal synthesis.

Abstract

Video-to-audio (V2A) generation leverages visual-only video features to render plausible sounds that match the scene. Importantly, the generated sound onsets should match the visual actions that are aligned with them, otherwise unnatural synchronization artifacts arise. Recent works have explored the progression of conditioning sound generators on still images and then video features, focusing on quality and semantic matching while ignoring synchronization, or by sacrificing some amount of quality to focus on improving synchronization only. In this work, we propose a V2A generative model, named MaskVAT, that interconnects a full-band high-quality general audio codec with a sequence-to-sequence masked generative model. This combination allows modeling both high audio quality, semantic matching, and temporal synchronicity at the same time. Our results show that, by combining a high-quality codec with the proper pre-trained audio-visual features and a sequence-to-sequence parallel structure, we are able to yield highly synchronized results on one hand, whilst being competitive with the state of the art of non-codec generative audio models. Sample videos and generated audios are available at https://maskvat.github.io .
Paper Structure (20 sections, 3 equations, 2 figures, 3 tables)

This paper contains 20 sections, 3 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Video to Audio Generation
  4. Audio-Visual Alignment Representations
  5. Autoregressive and Mask-based Audio Token Generation
  6. Method
  7. Audio Tokenizer
  8. Masked Generative Video-to-Audio Transformer
  9. Visual Conditioning
  10. Training Setup
  11. Sampling
  12. Beam-based selection
  13. Experiments
  14. Datasets
  15. Baselines
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Overview of the three main MaskVAT structures proposed.
  • Figure 2: Overview of the Training, Sampling and Selection parts involved in the MaskVAT framework.