PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers

TL;DR

This work tackles knowledge-intensive visual question answering by integrating retrieval with multi-modal models. It introduces M2KR, a diverse nine-dataset benchmark for image-, text-, and image+question-to-text retrieval, and PreFLMR, a pre-trained, fine-grained late-interaction multi-modal retriever built on an extended FLMR architecture. Through a four-stage training pipeline, including a substantial intermediate KB-VQA pre-training phase, PreFLMR achieves state-of-the-art retrieval performance across the M2KR tasks and substantially improves retrieval-augmented VQA results on OKVQA, Infoseek, and E-VQA. The study provides detailed insights into scaling vision and text encoders, the value of instruction-based prompts, and the benefits of domain-aligned intermediate pre-training for general-purpose multi-modal knowledge retrieval.

Abstract

Large Multimodal Models (LMMs) excel in natural language and visual understanding but are challenged by exacting tasks such as Knowledge-based Visual Question Answering (KB-VQA) which involve the retrieval of relevant information from document collections to use in shaping answers to questions. We present an extensive training and evaluation framework, M2KR, for KB-VQA. M2KR contains a collection of vision and language tasks which we have incorporated into a single suite of benchmark tasks for training and evaluating general-purpose multi-modal retrievers. We use M2KR to develop PreFLMR, a pre-trained version of the recently developed Fine-grained Late-interaction Multi-modal Retriever (FLMR) approach to KB-VQA, and we report new state-of-the-art results across a range of tasks. We also present investigations into the scaling behaviors of PreFLMR intended to be useful in future developments in general-purpose multi-modal retrievers.

Figures (4)

• Figure 1: PreFLMR Model Architecture. (1) the text query consists of an instruction and a question, which is encoded by a text encoder; (2) at the output of the vision encoder, a mapping network consisting of Multi-Layer Perceptrons (MLP) converts the '[CLS]' token representations into the same embedding space as the text encoder; (3) the transformer blocks take in the patch image embeddings from the penultimate layer of the vision encoder and attend to the text features by cross-attention; (4) a text encoder encodes documents in the knowledge base; (5) the scores between queries and documents are computed based on late-interaction, allowing each query token to interact with all document token embeddings.
• Figure 2: Change in Stage 3 validation loss when initialized from Stage 2 checkpoints after $N_{inter}$ steps of intermediate pre-training. A large difference indicates a greater gain from intermediate pre-training.
• Figure 3: Text encoder pre-training results evaluated on the full MSMARCO test set.
• Figure 3: PreFLMR achieves strong performance on the M2KR benchmark. The scale of the plot is adjusted for better visualization. The best and worst numbers of each task are annotated.