Learning Audio-Visual Embeddings with Inferred Latent Interaction Graphs

TL;DR

This work tackles the challenge of learning robust audio-visual embeddings under sparse annotations by separating meaningful semantic co-occurrences from incidental background signals. The authors introduce AV-SAL, a teacher that produces calibrated soft-label distributions; an inferred latent interaction (ILI) graph via the GRaSP algorithm; and a Latent Interaction Regularizer (LIR) that guides a student network to respect dependency-linked cross-modal pairs. Empirically, the approach yields consistent mean average precision improvements of about $1.5\%$ on AVE and VEGAS, demonstrating enhanced semantic coherence and robustness in cross-modal retrieval. The framework combines soft-label supervision with latent interaction inference to better reflect real-world co-occurrences, offering practical benefits for audio-visual understanding in noisy, multi-event scenes.

Abstract

Learning robust audio-visual embeddings requires bringing genuinely related audio and visual signals together while filtering out incidental co-occurrences - background noise, unrelated elements, or unannotated events. Most contrastive and triplet-loss methods use sparse annotated labels per clip and treat any co-occurrence as semantic similarity. For example, a video labeled "train" might also contain motorcycle audio and visual, because "motorcycle" is not the chosen annotation; standard methods treat these co-occurrences as negatives to true motorcycle anchors elsewhere, creating false negatives and missing true cross-modal dependencies. We propose a framework that leverages soft-label predictions and inferred latent interactions to address these issues: (1) Audio-Visual Semantic Alignment Loss (AV-SAL) trains a teacher network to produce aligned soft-label distributions across modalities, assigning nonzero probability to co-occurring but unannotated events and enriching the supervision signal. (2) Inferred Latent Interaction Graph (ILI) applies the GRaSP algorithm to teacher soft labels to infer a sparse, directed dependency graph among classes. This graph highlights directional dependencies (e.g., "Train (visual)" -> "Motorcycle (audio)") that expose likely semantic or conditional relationships between classes; these are interpreted as estimated dependency patterns. (3) Latent Interaction Regularizer (LIR): A student network is trained with both metric loss and a regularizer guided by the ILI graph, pulling together embeddings of dependency-linked but unlabeled pairs in proportion to their soft-label probabilities. Experiments on AVE and VEGAS benchmarks show consistent improvements in mean average precision (mAP), demonstrating that integrating inferred latent interactions into embedding learning enhances robustness and semantic coherence.

Figures (5)

• Figure 1: Overview of our architecture. Audio and visual features are extracted using VGGish and Inception V3. Logits from the audio and visual encoders are optimized with AV-SAL combined with a proxy-triplet loss (incorporating cross-modal attention) to produce calibrated soft-label distributions. These soft labels are input to GRaSP to infer a directed latent interaction graph $\widehat{A}$, where each edge $s_{a,i} \rightarrow s_{v,j}$ represents a dependency between audio label $i$ and visual label $j$. Finally, the embeddings are refined with a LIR term—weighted and normalized according to the discovered dependencies.
• Figure 2: Latent interaction edge frequency heatmap (on AVE). Each cell shows how often class $i$ was inferred to influence class $j$ across seven checkpoints.
• Figure 3: Latent interaction subgraphs from the AVE dataset across training epochs (400–900). The top row shows subgraphs centered on “Motorcycle (audio)” as the cause, the middle row on “Train horn (visual)” as the cause, and the bottom row on “Bus (visual)” as the effect. Blue nodes denote audio classes, and green nodes denote visual classes.
• Figure 4: Training loss (left) and test MAP (right) on AVE for different latent-interaction insertion epochs $M\in\{300,400,\dots,900\}$ (baseline = no insertion).
• Figure 5: Qualitative audio-to-visual retrieval on AVE. For two audio queries with different labels, the top-14 retrieved visuals are shown. Correct matches are highlighted in green, incorrect ones in red.