Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Video Object Segmentation-Aware Audio Generation

Ilpo Viertola, Vladimir Iashin, Esa Rahtu

TL;DR

This work defines video object segmentation-aware audio generation, enabling audio synthesis conditioned on object-level segmentation masks to achieve precise, visually localized Foley. It introduces SAGANet, a segmentation-aware extension of a diffusion-based multimodal audio generator that fuses global and focal visual cues via a focal prompt and gated cross-attention, with optional LoRA fine-tuning. To support research in segmentation-aware Foley, the authors present Segmented Music Solos, a pipeline and dataset of solo-instrument videos with sounding-object masks, combined with a verification pipeline and mask-generation strategy. Experimental results show that SAGANet improves semantic alignment and temporal synchronization, particularly in multi-source scenes, and demonstrate strong generalization from single-source training to multi-source test cases. Overall, the work provides a foundation for controllable, high-fidelity Foley synthesis and a dataset to study segmentation-aware audio generation.

Abstract

Existing multimodal audio generation models often lack precise user control, which limits their applicability in professional Foley workflows. In particular, these models focus on the entire video and do not provide precise methods for prioritizing a specific object within a scene, generating unnecessary background sounds, or focusing on the wrong objects. To address this gap, we introduce the novel task of video object segmentation-aware audio generation, which explicitly conditions sound synthesis on object-level segmentation maps. We present SAGANet, a new multimodal generative model that enables controllable audio generation by leveraging visual segmentation masks along with video and textual cues. Our model provides users with fine-grained and visually localized control over audio generation. To support this task and further research on segmentation-aware Foley, we propose Segmented Music Solos, a benchmark dataset of musical instrument performance videos with segmentation information. Our method demonstrates substantial improvements over current state-of-the-art methods and sets a new standard for controllable, high-fidelity Foley synthesis. Code, samples, and Segmented Music Solos are available at https://saganet.notion.site

Video Object Segmentation-Aware Audio Generation

TL;DR

This work defines video object segmentation-aware audio generation, enabling audio synthesis conditioned on object-level segmentation masks to achieve precise, visually localized Foley. It introduces SAGANet, a segmentation-aware extension of a diffusion-based multimodal audio generator that fuses global and focal visual cues via a focal prompt and gated cross-attention, with optional LoRA fine-tuning. To support research in segmentation-aware Foley, the authors present Segmented Music Solos, a pipeline and dataset of solo-instrument videos with sounding-object masks, combined with a verification pipeline and mask-generation strategy. Experimental results show that SAGANet improves semantic alignment and temporal synchronization, particularly in multi-source scenes, and demonstrate strong generalization from single-source training to multi-source test cases. Overall, the work provides a foundation for controllable, high-fidelity Foley synthesis and a dataset to study segmentation-aware audio generation.

Abstract

Existing multimodal audio generation models often lack precise user control, which limits their applicability in professional Foley workflows. In particular, these models focus on the entire video and do not provide precise methods for prioritizing a specific object within a scene, generating unnecessary background sounds, or focusing on the wrong objects. To address this gap, we introduce the novel task of video object segmentation-aware audio generation, which explicitly conditions sound synthesis on object-level segmentation maps. We present SAGANet, a new multimodal generative model that enables controllable audio generation by leveraging visual segmentation masks along with video and textual cues. Our model provides users with fine-grained and visually localized control over audio generation. To support this task and further research on segmentation-aware Foley, we propose Segmented Music Solos, a benchmark dataset of musical instrument performance videos with segmentation information. Our method demonstrates substantial improvements over current state-of-the-art methods and sets a new standard for controllable, high-fidelity Foley synthesis. Code, samples, and Segmented Music Solos are available at https://saganet.notion.site

Paper Structure

This paper contains 30 sections, 3 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Video Object Segmentation
  4. Artificial Foley Models
  5. Controlled Generation with Pretrained Audio Networks
  6. Method
  7. Video Object Segmentation-aware Audio Generation Task Formulation
  8. SAGANet Architecture
  9. Focal Prompt.
  10. Localized Vision Backbone with Temporal Mask Embedding.
  11. Audio Generation.
  12. Segmented Music Solos Dataset
  13. Stage 1: Source Videos.
  14. Stage 2: Visual Verification.
  15. Stage 3: Auditory Verification.
  16. ...and 15 more sections

Figures (2)

  • Figure 1: Overview of SAGANet control module. Given a video and its corresponding segmentation masks, the model combines global and local information streams. Gated Cross-Attention layers alayrac2022flamingoli2022blip are used to fuse global and local features extracted by Synchformer iashin2024synchformer, with shared weights across both branches. Only the layers highlighted in orange are updated during training. The final audio is generated following the same procedure as in the base MMAudio model. For additional details on MMAudio, refer to chengTamingMultimodalJoint2024.
  • Figure 2: Samples from Segmented Music Solos. The top row indicates the musical instrument label. The second row displays the first frames of the video and the corresponding mask stream. Last row displays the audio associated with the sample.