Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RAMQA: A Unified Framework for Retrieval-Augmented Multi-Modal Question Answering

Yang Bai, Christan Earl Grant, Daisy Zhe Wang

TL;DR

RAMQA tackles the gap between encoder-based ranking and decoder-based multi-modal LLMs in multi-modal QA by introducing a two-stage retrieval framework. It first uses RankLLaVA for pointwise multi-modal ranking, then applies RAMLLaMA, an autoregressive, instruction-tuned generator, to re-rank top-k candidates and produce answers, aided by zero-shot image-to-text data unification. The approach, validated on WebQA and MultimodalQA, yields state-of-the-art QA scores and EM/F1 improvements, driven by permutation-based generative retrieval and a multi-task objective. This work advances practical multi-modal IR by enabling end-to-end, efficient retrieval and generation with modern LLMs and provides a blueprint for integrating LTR and generative retrieval in multi-modal contexts.

Abstract

Multi-modal retrieval-augmented Question Answering (MRAQA), integrating text and images, has gained significant attention in information retrieval (IR) and natural language processing (NLP). Traditional ranking methods rely on small encoder-based language models, which are incompatible with modern decoder-based generative large language models (LLMs) that have advanced various NLP tasks. To bridge this gap, we propose RAMQA, a unified framework combining learning-to-rank methods with generative permutation-enhanced ranking techniques. We first train a pointwise multi-modal ranker using LLaVA as the backbone. Then, we apply instruction tuning to train a LLaMA model for re-ranking the top-k documents using an innovative autoregressive multi-task learning approach. Our generative ranking model generates re-ranked document IDs and specific answers from document candidates in various permutations. Experiments on two MRAQA benchmarks, WebQA and MultiModalQA, show significant improvements over strong baselines, highlighting the effectiveness of our approach. Code and data are available at: https://github.com/TonyBY/RAMQA

RAMQA: A Unified Framework for Retrieval-Augmented Multi-Modal Question Answering

TL;DR

RAMQA tackles the gap between encoder-based ranking and decoder-based multi-modal LLMs in multi-modal QA by introducing a two-stage retrieval framework. It first uses RankLLaVA for pointwise multi-modal ranking, then applies RAMLLaMA, an autoregressive, instruction-tuned generator, to re-rank top-k candidates and produce answers, aided by zero-shot image-to-text data unification. The approach, validated on WebQA and MultimodalQA, yields state-of-the-art QA scores and EM/F1 improvements, driven by permutation-based generative retrieval and a multi-task objective. This work advances practical multi-modal IR by enabling end-to-end, efficient retrieval and generation with modern LLMs and provides a blueprint for integrating LTR and generative retrieval in multi-modal contexts.

Abstract

Multi-modal retrieval-augmented Question Answering (MRAQA), integrating text and images, has gained significant attention in information retrieval (IR) and natural language processing (NLP). Traditional ranking methods rely on small encoder-based language models, which are incompatible with modern decoder-based generative large language models (LLMs) that have advanced various NLP tasks. To bridge this gap, we propose RAMQA, a unified framework combining learning-to-rank methods with generative permutation-enhanced ranking techniques. We first train a pointwise multi-modal ranker using LLaVA as the backbone. Then, we apply instruction tuning to train a LLaMA model for re-ranking the top-k documents using an innovative autoregressive multi-task learning approach. Our generative ranking model generates re-ranked document IDs and specific answers from document candidates in various permutations. Experiments on two MRAQA benchmarks, WebQA and MultiModalQA, show significant improvements over strong baselines, highlighting the effectiveness of our approach. Code and data are available at: https://github.com/TonyBY/RAMQA
Paper Structure (33 sections, 8 figures, 4 tables, 2 algorithms)

This paper contains 33 sections, 8 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-Modal Retrieval-Augmented Question Answering
  4. Learning-to-Rank
  5. Generative Retrieval
  6. Methodology
  7. Preliminaries
  8. Task Definition
  9. LLaMA
  10. LLaVA
  11. RankLLaVA for Multi-modal Pointwise Ranking
  12. Multi-task Generation
  13. Data Unification
  14. RAMLLaMA
  15. Experiments
  16. ...and 18 more sections

Figures (8)

  • Figure 1: An example in WebQA Chang2021WebQAMA, a Multi-modal Open-domain Question-Answering benchmark. This task requires the system to precisely identify critical sources from distractors and use these key sources to infer the answers.
  • Figure 2: RAMQA Framework Overview. A detailed description of the three main components—RankLLaVA, Data Unification (Image to Text), and RAMLLaMA—is provided in Sections \ref{['sec: rankllava']}, \ref{['sec: data_unification']}, and \ref{['sec: RAMLLaMA']}, respectively.
  • Figure 3: Zero-shot image description generation for data modality unification.
  • Figure 4: Impact of Input Length on RAMLLaMA Performance on the WebQA test set. The horizontal axis(#Doc) represents the number of candidate documents from RankLLaVA's output included in RAMLLaMA's input prompt during both training and testing. We ensured that the input prompt length did not exceed LLaMA3's limit of 8096 tokens in any of the experiments.
  • Figure 5: A Training data example of RAMLLaMA.
  • ...and 3 more figures