BrewCLIP: A Bifurcated Representation Learning Framework for Audio-Visual Retrieval

TL;DR

BrewCLIP tackles audio-image retrieval by introducing a bifurcated representation that preserves both textual information and non-textual audio cues. Built on a dual-channel architecture that combines a transcription-based CLIP pipeline (Whisper + CLIP) with an End-to-End audio encoder, and augmented by shared prompting, BrewCLIP achieves state-of-the-art performance across multiple datasets, including scripted and unscripted speech. The approach demonstrates that non-textual information such as mood can be learned from audio representations, and it validates the benefits of prompt-based fine-tuning for cross-modal alignment. Overall, BrewCLIP delivers a robust, scalable joint audio-visual representation, offering practical gains for retrieval and potential extensions to mood-aware and more nuanced audio-visual tasks.

Abstract

Previous methods for audio-image matching generally fall into one of two categories: pipeline models or End-to-End models. Pipeline models first transcribe speech and then encode the resulting text; End-to-End models encode speech directly. Generally, pipeline models outperform end-to-end models, but the intermediate transcription necessarily discards some potentially useful non-textual information. In addition to textual information, speech can convey details such as accent, mood, and and emphasis, which should be effectively captured in the encoded representation. In this paper, we investigate whether non-textual information, which is overlooked by pipeline-based models, can be leveraged to improve speech-image matching performance. We thoroughly analyze and compare End-to-End models, pipeline models, and our proposed dual-channel model for robust audio-image retrieval on a variety of datasets. Our approach achieves a substantial performance gain over the previous state-of-the-art by leveraging strong pretrained models, a prompting mechanism and a bifurcated design.

Figures (9)

• Figure 1: Venn diagram illustrating our dual-channel design. Our text channel primarily focuses on capturing textual information, while audio channel can complements the textual information and facilitates the communication of non-textual information.
• Figure 2: Detailed diagram of our proposed model. The acoustic channel is shaded in yellow and the transcription based pipeline channel is shaded in red. E2E-only model only contains the Acoustic branch and image branch. PIP-Only model only contains the pipeline branch and image branch. PIP-only(Zero-shot) is a special pipeline-based model which is not finetuned by the prompts. black Sun: frozen, white Sun: updating.
• Figure 3: Qualitative analysis of sample difference between LN-COCO and SpokenCOCO and their impacts on model performance. Audio descriptions from Flick8K and SpokenCOCO typically consist of concise, pre-planned single sentences. Such scripted descriptions pose for the recall task as they might match with multiple similar images. The LN-COCO samples contains much more detailed information to help the model localize the exact match of the image. In the Audio2Image matching task, Similarity Rank refers the rank of the current image relative to all provided images,determined on Cosine similarity score and the Best-Match Image is the one that has the highest similarity score to the given audio. (More examples in Appendix)
• Figure 4: Qualitative analysis of different variations of models on audio to image retrieval. The top case shows our full model can attend to non-major content, making optimal use of the rich information in the LN-COCO samples, whereas E2E model and Zero-Shot pipeline model focus solely on the major components. The bottom case shows our full model can still successfully retrieve the correct image, even the transcription has a major mistake, underscoring the necessity of the inclusion of the E2E channel.(More examples in Appendix)
• Figure 5: Failed case due to the Image Center Crop transform. The original image features a clock, a detail referenced in the spoken expression; however, the clock is cropped in the image.