Conditional Flow Matching for Visually-Guided Acoustic Highlighting

TL;DR

Visually-guided Acoustic Highlighting is reformulated as a conditional, flow-based remixing problem to address the many-to-many nature of audio balancing in videos. The authors introduce Conditional Flow Matching (CFM) with a rollout loss and an enhanced conditioning module that fuses audio into the visual encoder for early cross-modal source selection. Empirical results on the Muddy Mix dataset show VisAH-FM consistently outperforms prior discriminative approaches across signal and semantic metrics, with ablations confirming the critical roles of rollout and audio-conditioned conditioning. This work demonstrates that generative, flow-based models can achieve more coherent audio-visual alignment, enabling stable long-range generation and suggesting avenues for unpaired data training in the future.

Abstract

Visually-guided acoustic highlighting seeks to rebalance audio in alignment with the accompanying video, creating a coherent audio-visual experience. While visual saliency and enhancement have been widely studied, acoustic highlighting remains underexplored, often leading to misalignment between visual and auditory focus. Existing approaches use discriminative models, which struggle with the inherent ambiguity in audio remixing, where no natural one-to-one mapping exists between poorly-balanced and well-balanced audio mixes. To address this limitation, we reframe this task as a generative problem and introduce a Conditional Flow Matching (CFM) framework. A key challenge in iterative flow-based generation is that early prediction errors -- in selecting the correct source to enhance -- compound over steps and push trajectories off-manifold. To address this, we introduce a rollout loss that penalizes drift at the final step, encouraging self-correcting trajectories and stabilizing long-range flow integration. We further propose a conditioning module that fuses audio and visual cues before vector field regression, enabling explicit cross-modal source selection. Extensive quantitative and qualitative evaluations show that our method consistently surpasses the previous state-of-the-art discriminative approach, establishing that visually-guided audio remixing is best addressed through generative modeling.

Figures (8)

• Figure 1: We propose to cast visually-guided acoustic highlighting in the flow matching setup. Acoustic sources are iteratively highlighted or not based on the visual context. This simplified illustration shows that when audio and visual cues are opposed, the source is subdued. However, when they showcase the same event, the source is enhanced.
• Figure 2: Illustration of rollout loss. Unlike the regular flow matching loss, which always uses a ground truth trajectory point as input, the rollout loss applies the loss after performing all the steps, forcing the model to have a coherent global trajectory to avoid compounding errors.
• Figure 3: Overview of VisAH-FM architecture: Building upon the VisAH visah backbone (purple box), VisAH-FM introduces time step conditioning, along with a novel rollout loss to guide flow matching training. Furthermore, we also incorporate an improved multimodal conditioning module (green box) where audio information is injected into intermediate CLIP representation through cross attention.
• Figure 4: Three-fold analysis of the impact of the rollout loss. Combined with the flow matching loss, the rollout allows stable trajectory prediction and avoids error accumulation. When used without flow matching loss, it learns nonlinear trajectories.
• Figure 5: Illustration of the behavior of the model trained with and without rollout loss. Waveforms are overlapped for readability. The first step is similar, but the difference appears in the latter steps when the rollout-trained model enhances speech, while the other does not.