X-Aligner: Composed Visual Retrieval without the Bells and Whistles

TL;DR

The paper addresses Composed Video Retrieval by showing that single-stage fusion of multimodal queries is insufficient. It introduces X-Aligner, a multi-stage cross-attention fusion module that also leverages a visual caption, combined with a two-stage fine-tuning protocol on pretrained Vision-Language Models (BLIP/BLIP-2). The approach achieves state-of-the-art Recall@1 on WebVid-CoVR-Test ($R@1=63.93\%$) and demonstrates strong zero-shot CIR generalization to FashionIQ and CIRCO, with ablations highlighting the benefits of caption inputs and guided text encoder adaptation. The work suggests that lightweight, progressively fused multimodal representations can surpass heavier models and offers practical insights for cross-domain retrieval tasks, while noting reliance on external captioning models as a potential limitation.

Abstract

Composed Video Retrieval (CoVR) facilitates video retrieval by combining visual and textual queries. However, existing CoVR frameworks typically fuse multimodal inputs in a single stage, achieving only marginal gains over initial baseline. To address this, we propose a novel CoVR framework that leverages the representational power of Vision Language Models (VLMs). Our framework incorporates a novel cross-attention module X-Aligner, composed of cross-attention layers that progressively fuse visual and textual inputs and align their multimodal representation with that of the target video. To further enhance the representation of the multimodal query, we incorporate the caption of the visual query as an additional input. The framework is trained in two stages to preserve the pretrained VLM representation. In the first stage, only the newly introduced module is trained, while in the second stage, the textual query encoder is also fine-tuned. We implement our framework on top of BLIP-family architecture, namely BLIP and BLIP-2, and train it on the Webvid-CoVR data set. In addition to in-domain evaluation on Webvid-CoVR-Test, we perform zero-shot evaluations on the Composed Image Retrieval (CIR) data sets CIRCO and Fashion-IQ. Our framework achieves state-of-the-art performance on CoVR obtaining a Recall@1 of 63.93% on Webvid-CoVR-Test, and demonstrates strong zero-shot generalization on CIR tasks.

Figures (5)

• Figure 1: We propose a novel framework that leverages multi-stage cross-attention and video captions to accurately align multimodal queries. As shown, the single-stage baseline CoVR-BLIP Ventura-AAAI-2024 fails interpret the "turn it into a male" instruction correctly, retrieving a video with a male subject. In contrast, our method fuses the visual, textual, and caption inputs to retrieve the correct video of a man painting. Our approach achieves state-of-the-art performance, such as a Recall@1 of 63.93% on Webvid-CoVR-Test.
• Figure 2: We present an overview of our framework. Our fusion adapter X-Aligner is integrated on top of the embeddings extracted from the Vision-Language Models. The Text Encoder and Query Text Encoder share the same parameters in Stage 1. The resulting multimodal embedding is then aligned with the target embeddings (visual and textual) using contrastive loss. Components updated during each training stage are indicated with dashed and dotted lines. "Tab" is the nickname used for tabby cat.
• Figure 3: We present the components of X-Aligner. We obtain the multimodal embedding $emb_{tv}$ by enriching the text embedding with information from the visual query ($emb_v$ and $emb_c$). We progressively integrate multimodal input by applying cross-attention between the text embedding $emb_t$ and the multimodal embedding $emb_{vt}$. The final embedding $emb_{mm}$ is computed as the average of the embeddings produced by each components, namely $emb_{tv}$ and $emb_{tv'}$. We depict with black arrows the query, and with red arrows the keys and values.
• Figure 4: Visual query samples along with their original captions provided by Ventura et al. Ventura-AAAI-2024 and captions generated by InternVL Chen-CVPR-2024. It is noticeable that captions generated by InternVL Chen-CVPR-2024 are more closely aligned with the visual inputs compared to the original ones.
• Figure 5: Qualitative results comparing the results obtained using our framework and CoVR-BLIP Ventura-AAAI-2024. Our framework is able to retrieve the correct target given the multimodal input. The samples are extracted from WebVid-CoVR-Test Ventura-AAAI-2024 and we display only the middle frame of the videos for clarity.