Learning to Highlight Audio by Watching Movies

TL;DR

We address visually-guided acoustic highlighting by learning a mapping from poorly mixed audio and video context to a highlighted audio output. The proposed VisAH model combines a dual U-Net audio backbone with a latent highlighting transformer that fuses video and optional text captions via cross-attention to reweight audio sources. Training leverages free supervision from movies through a three-step muddy mix data generation process (separation, adjustment, remix) applied to the Condensed Movie Dataset, enabling realistic, varied inputs. Empirical results show VisAH outperforms baselines on waveform fidelity, semantic alignment, and audio-visual saliency, with subjective tests corroborating perceptual quality improvements. The work offers a scalable path to harmonize audio with visual narratives and points to further multimodal fusion and data-generation refinements as future directions.

Abstract

Recent years have seen a significant increase in video content creation and consumption. Crafting engaging content requires the careful curation of both visual and audio elements. While visual cue curation, through techniques like optimal viewpoint selection or post-editing, has been central to media production, its natural counterpart, audio, has not undergone equivalent advancements. This often results in a disconnect between visual and acoustic saliency. To bridge this gap, we introduce a novel task: visually-guided acoustic highlighting, which aims to transform audio to deliver appropriate highlighting effects guided by the accompanying video, ultimately creating a more harmonious audio-visual experience. We propose a flexible, transformer-based multimodal framework to solve this task. To train our model, we also introduce a new dataset -- the muddy mix dataset, leveraging the meticulous audio and video crafting found in movies, which provides a form of free supervision. We develop a pseudo-data generation process to simulate poorly mixed audio, mimicking real-world scenarios through a three-step process -- separation, adjustment, and remixing. Our approach consistently outperforms several baselines in both quantitative and subjective evaluation. We also systematically study the impact of different types of contextual guidance and difficulty levels of the dataset. Our project page is here: https://wikichao.github.io/VisAH/.

Figures (12)

• Figure 1: We propose a new task that aims to transform poorly mixed audio into a well-balanced mix using visual guidance. One of our key insights is to use well-curated audio-visual content from a movie database as free supervision to learn the appropriate highlighting effect for audio (L2H).
• Figure 2: Overview of VisAH: (a) Our model takes a poorly mixed audio waveform as input and produces the highlighted audio using a dual U-Net architecture. For simplicity, skip connections are omitted in the illustration. (b) The latent highlighting transformer incorporates vision and text encoders to integrate temporal information, guiding the transformer decoder to transform audio features effectively.
• Figure 3: We generate poorly mixed audio from the well-mixed movie audio through the following steps: 1) Separation: We separate the ground truth movie audio into individual tracks for speech, music, and sound effects, allowing for some imperfections in the separation process; 2) Adjustment: For each separated track, we apply either suppression or emphasis, with the intensity selected from three levels: [high, moderate, low]; 3) Remixing: Finally, we combine the adjusted tracks through simple addition to create the poorly mixed input audio.
• Figure 4: We perform a qualitative comparison by visualizing the waveform and magnitude spectrograms of the highlighted audio results from different methods, along with the input and ground truth. Our method produces results that are closest to the movie GT. The orange box denotes suppressed snippets, and green box indicates highlighted snippets.
• Figure 5: Subjective test: We ask users to rank the four methods based on audio-visual balance to evaluate acoustic highlighting.