Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RAVEN: Multitask Retrieval Augmented Vision-Language Learning

Varun Nagaraj Rao, Siddharth Choudhary, Aditya Deshpande, Ravi Kumar Satzoda, Srikar Appalaraju

TL;DR

AVEN is introduced, a multitask retrieval augmented VLM framework that enhances base VLMs through efficient, task specific fine-tuning and demonstrates the efficacy of applying RAG approaches to VLMs, marking a stride toward more efficient and accessible multimodal learning.

Abstract

The scaling of large language models to encode all the world's knowledge in model parameters is unsustainable and has exacerbated resource barriers. Retrieval-Augmented Generation (RAG) presents a potential solution, yet its application to vision-language models (VLMs) is under explored. Existing methods focus on models designed for single tasks. Furthermore, they're limited by the need for resource intensive pre training, additional parameter requirements, unaddressed modality prioritization and lack of clear benefit over non-retrieval baselines. This paper introduces RAVEN, a multitask retrieval augmented VLM framework that enhances base VLMs through efficient, task specific fine-tuning. By integrating retrieval augmented samples without the need for additional retrieval-specific parameters, we show that the model acquires retrieval properties that are effective across multiple tasks. Our results and extensive ablations across retrieved modalities for the image captioning and VQA tasks indicate significant performance improvements compared to non retrieved baselines +1 CIDEr on MSCOCO, +4 CIDEr on NoCaps and nearly a +3\% accuracy on specific VQA question types. This underscores the efficacy of applying RAG approaches to VLMs, marking a stride toward more efficient and accessible multimodal learning.

RAVEN: Multitask Retrieval Augmented Vision-Language Learning

TL;DR

AVEN is introduced, a multitask retrieval augmented VLM framework that enhances base VLMs through efficient, task specific fine-tuning and demonstrates the efficacy of applying RAG approaches to VLMs, marking a stride toward more efficient and accessible multimodal learning.

Abstract

The scaling of large language models to encode all the world's knowledge in model parameters is unsustainable and has exacerbated resource barriers. Retrieval-Augmented Generation (RAG) presents a potential solution, yet its application to vision-language models (VLMs) is under explored. Existing methods focus on models designed for single tasks. Furthermore, they're limited by the need for resource intensive pre training, additional parameter requirements, unaddressed modality prioritization and lack of clear benefit over non-retrieval baselines. This paper introduces RAVEN, a multitask retrieval augmented VLM framework that enhances base VLMs through efficient, task specific fine-tuning. By integrating retrieval augmented samples without the need for additional retrieval-specific parameters, we show that the model acquires retrieval properties that are effective across multiple tasks. Our results and extensive ablations across retrieved modalities for the image captioning and VQA tasks indicate significant performance improvements compared to non retrieved baselines +1 CIDEr on MSCOCO, +4 CIDEr on NoCaps and nearly a +3\% accuracy on specific VQA question types. This underscores the efficacy of applying RAG approaches to VLMs, marking a stride toward more efficient and accessible multimodal learning.
Paper Structure (23 sections, 1 equation, 5 figures, 4 tables)

This paper contains 23 sections, 1 equation, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision Language Models
  4. Retrieval Augmented Generation in NLP
  5. Retrieval Augmented Generation in VLMs
  6. Proposed Approach
  7. RAVEN Framework
  8. Multimodal Retriever
  9. Base Vision-Language Model (VLM)
  10. Experiments
  11. Training Setup
  12. Data
  13. Missing Samples:
  14. Implementation
  15. Evaluation Setup
  16. ...and 8 more sections

Figures (5)

  • Figure 1: Illustration of our RAVEN framework. Given an input image, we retrieve image-text pairs from an external memory. Subsequently, we use a multitask pretrained base vision-language model (VLM) to encode the retrieved samples along with the query and decode to generate an output by attending over both the query and retrieved samples.
  • Figure 2: Examples of the retriever output given a query image.
  • Figure 3: Examples where RAVEN succeeds in generating the correct answer.
  • Figure 4: Examples where RAVEN fails in generating the correct answer.
  • Figure 5: An example depicting the benefits of using a multimodal query embedding (average of image and question embedding). This results in the retrieval of captions relevant to both the image and question.