LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant

TL;DR

LamRA reframes multimodal retrieval as a universal task for Large Multimodal Models by inserting lightweight LoRA adapters, enabling a two-stage retrieval training and a joint reranking module. It demonstrates robust performance across more than ten retrieval tasks, with strong zero-shot generalization to unseen datasets and tasks, surpassing dual-encoder baselines in many cross-modal scenarios. The framework combines LamRA-Ret for initial retrieval and LamRA-Rank for reranking, producing a final score that balances embedding similarity with learned rank signals. Together, these contributions suggest that Large Multimodal Models can serve as universal retrievers and rerankers, offering practical improvements for diverse multimodal information retrieval applications.

Abstract

With the rapid advancement of multimodal information retrieval, increasingly complex retrieval tasks have emerged. Existing methods predominately rely on task-specific fine-tuning of vision-language models, often those trained with image-text contrastive learning. In this paper, we explore the possibility of re-purposing generative Large Multimodal Models (LMMs) for retrieval. This approach enables unifying all retrieval tasks under the same formulation and, more importantly, allows for extrapolation towards unseen retrieval tasks without additional training. Our contributions can be summarised in the following aspects: (i) We introduce LamRA, a versatile framework designed to empower LMMs with sophisticated retrieval and reranking capabilities. (ii) For retrieval, we adopt a two-stage training strategy comprising language-only pre-training and multimodal instruction tuning to progressively enhance LMM's retrieval performance. (iii) For reranking, we employ joint training for both pointwise and listwise reranking, offering two distinct ways to further boost the retrieval performance. (iv) Extensive experimental results underscore the efficacy of our method in handling more than ten retrieval tasks, demonstrating robust performance in both supervised and zero-shot settings, including scenarios involving previously unseen retrieval tasks.

Figures (13)

• Figure 1: The LamRA framework empowers Large Multimodal Models with advanced retrieval and reranking capabilities. (a) LamRA enhances LMMs with universal retrieval and reranking capabilities by inserting lightweight LoRA modules into the LMMs. (b) Examples of varied retrieval tasks demonstrate LamRA's capability to handle diverse retrieval tasks. (c) Performance comparison on the M-BEIR test set shows LamRA's superior performance across a wide range of retrieval tasks. For instance, $q^t \to c^i$ represents text-to-image retrieval.
• Figure 2: Overview of the proposed LamRA framework. LamRA consists of two components: LamRA-Ret and LamRA-Rank. The top section illustrates LamRA-Ret, encompassing both the pre-training and instruction-tuning stages, where contrastive learning is employed to enhance the retrieval capability of LMMs. The pre-training stage aims to improve the feature extraction capabilities through text-to-text retrieval, while the instruction tuning stage adapts the LMMs to various retrieval tasks by fine-tuning on diverse tasks with task-specific instructions. The bottom section depicts the joint training process of LamRA-Rank, which integrates both pointwise and listwise reranking.
• Figure 3: Qualitative examples. We show the results of our method across six different retrieval tasks, with the ground truth indicated by a red box and red text. Here, $q^t$ for text queries, $q^i$ for image queries, $c^t$ for text candidates, $c^i$ for image candidates, and $c^v$ for video candidates. For more qualitative examples, please refer to Appendix \ref{['sec:supp_qualitative']}.
• Figure 4: Qualitative examples on text-to-image retrieval task, where the red box marks the ground truth.
• Figure 5: Qualitative examples on text-to-text retrieval task, where the red text marks the ground truth.