An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignment

TL;DR

This work tackles zero-shot audio captioning by bridging the audiovisual modality gap with Distribution ALIgnment (DALI), aligning the full token distributions of an audio backbone to an image-captioner’s space using two distribution-matching strategies: Maximum Mean Discrepancy (MMD) and Optimal Transport (OT), including an attentive OT variant. A two-stage framework combines prefix-tuning and audiovisual distillation to adapt a frozen image captioner (Llava 1.5) for audio captioning without sacrificing image performance, enabling either audio-only or audiovisual inputs. On AudioCaps and Clotho, DALI variants—especially the attentive OT with audiovisual input—achieve state-of-the-art or competitive zero-shot audio captioning results without annotated audio-caption pairs, illustrating strong cross-modal transfer and scalability. The approach highlights that modeling full distributions and learning cross-modal transport weights can mitigate the modality gap more effectively than contrastive means, offering a practical path to scalable multimodal captioning across audio and visual streams.

Abstract

Multimodal large language models have fueled progress in image captioning. These models, fine-tuned on vast image datasets, exhibit a deep understanding of semantic concepts. In this work, we show that this ability can be re-purposed for audio captioning, where the joint image-language decoder can be leveraged to describe auditory content associated with image sequences within videos featuring audiovisual content. This can be achieved via multimodal alignment. Yet, this multimodal alignment task is non-trivial due to the inherent disparity between audible and visible elements in real-world videos. Moreover, multimodal representation learning often relies on contrastive learning, facing the challenge of the so-called modality gap which hinders smooth integration between modalities. In this work, we introduce a novel methodology for bridging the audiovisual modality gap by matching the distributions of tokens produced by an audio backbone and those of an image captioner. Our approach aligns the audio token distribution with that of the image tokens, enabling the model to perform zero-shot audio captioning in an unsupervised fashion while keeping the initial image captioning component unaltered. This alignment allows for the use of either audio or audiovisual input by combining or substituting the image encoder with the aligned audio encoder. Our method achieves significantly improved performances in zero-shot audio captioning, compared to existing approaches.

Figures (8)

• Figure 1: Conventional audiovisual alignment through contrastive learning leads to a gap between modalities. Our proposed distribution alignment method matches closely both distributions leading to better joint representations for audio captioning.
• Figure 2: Overview of the proposed approach. In the first stage, a prefix tuning is performed using a few (image, audio-caption) pairs (1-a). Additionally, the audio backbone is aligned with the image backbone (1-b) through distribution alignment. Audio captioning can then be performed by switching the image backbone with the audio backbone and adding the prefix tokens (1-c). In a second stage, visually-informed audio captions are generated using both audio, image, and prefix tokens. The MLP mapping the audio encoder to the language model is then fine-tuned with these pseudo captions (2-d). The final inference for audio captioning, using audio or audio visual inputs, is performed by forwarding the aligned audio backbone's output through the trained MLP to obtain the LLM input (2-e).
• Figure 3: Multimodal distribution alignment through optimal transport. The audio and image tokens are used to compute the cost matrix, while two separate cross-attention layers estimate the weights $\alpha^{\text{Att}}$ and $\beta^{\text{Att}}$.
• Figure 4: AudioCaps average tokens distribution. While contrastive learning maps the audio in a space separate from the image ones, MMD and optimal transport project in the same part of the space. The model trained using attentive optimal transport projects the audios in a space closer to the image, with marginal overlap.
• Figure 5: Captioning performances according to the number of image captions. After 16, the performance does not improve, indicating that the prefix tuning does not play an important role in the learning process, it just specifies the task.