Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Language

TL;DR

DenseAV tackles the problem of grounding language and sounds to visual objects without supervision by learning high-resolution audio-visual representations through self-supervised contrastive learning. It introduces a dual-encoder architecture with a novel multi-head similarity aggregator that produces a dense, semantically meaningful AV activation volume, enabling both speech- and sound-grounded semantic segmentation. The model automatically disentangles language from sound using a disentanglement regularizer, and achieves state-of-the-art performance on cross-modal retrieval and dense grounding benchmarks, outperforming prior methods with fewer parameters. This approach offers a scalable pathway to fine-grained, zero-shot visual grounding of spoken language and environmental sounds with practical implications for multimodal understanding in zero-annotation regimes.

Abstract

We present DenseAV, a novel dual encoder grounding architecture that learns high-resolution, semantically meaningful, and audio-visually aligned features solely through watching videos. We show that DenseAV can discover the ``meaning'' of words and the ``location'' of sounds without explicit localization supervision. Furthermore, it automatically discovers and distinguishes between these two types of associations without supervision. We show that DenseAV's localization abilities arise from a new multi-head feature aggregation operator that directly compares dense image and audio representations for contrastive learning. In contrast, many other systems that learn ``global'' audio and video representations cannot localize words and sound. Finally, we contribute two new datasets to improve the evaluation of AV representations through speech and sound prompted semantic segmentation. On these and other datasets we show DenseAV dramatically outperforms the prior art on speech and sound prompted semantic segmentation. DenseAV outperforms the previous state-of-the-art, ImageBind, on cross-modal retrieval using fewer than half of the parameters. Project Page: \href{https://aka.ms/denseav}{https://aka.ms/denseav}

Figures (10)

• Figure 1: Visual overview of the DenseAV algorithm. Two modality-specific backbones featurize audio and visual signals. We introduce a novel generalization of multi-head attention to extract attention maps that discover and separate the "meaning" of spoken words and the sounds an object makes. DenseAV performs this localization and decomposition solely through observing paired stimuli such as videos.
• Figure 2: Qualitative comparison of several modern architectures for associating audio and video modalities. Only DenseAV learns a high-resolution and semantically aligned set of local features. This allows us to perform speech and sound prompted semantic segmentation using only the inner products between deep features. Other approaches, such as ImageBind, do not show aligned local feature maps. Approaches that do show some localization capabilities, like DAVENet, do not generalize to sound and language, and do not achieve the high-resolution localization capabilities of DenseAV. Dense features are visualized using PCA as in hamilton2022unsupervised
• Figure 3: Architectural overview of our multi-head attention aggregator. Dense feature maps are split into $K$ heads $(K=1,2)$ in our experiments. We form an AV activation tensor by taking the inner-products of each head's features across the spatial and temporal extent of the visual and audio signals respectively as in Equation \ref{['eqn:pairedsim']}. We then aggregate this similarity volume into a single similarity score by max-pooling head and spatial dimensions and average-pooling audio dimensions. Our approach aims to encourage the network to identify specific shared objects between the audio and visual modalities. In particular, max-pooling of heads disentangles sound and language, and max-pooling spatial dimensions helps localize objects.
• Figure 4: Selected visualizations of AV heatmaps for the speech prompted semantic segmentation task. We visualize results across several baselines. DenseAV achieves the best localization performance both qualitatively and quantitatively, highlighting the full extent of objects with high resolution heatmaps.
• Figure 5: Selected visualizations of AV heatmaps for the sound prompted semantic segmentation task. We visualize results across several baselines. DenseAV achieves the best localization performance both qualitatively and quantitatively, highlighting the full extent of objects with high resolution heatmaps. We note that DenseAV can highlight objects even if they are not centered or clearly visible as in the dog example (second column).