MAVIS: A Benchmark for Multimodal Source Attribution in Long-form Visual Question Answering

TL;DR

MAVIS introduces a large-scale benchmark for multimodal source attribution in open-domain, long-form visual question answering, pairing visual questions with sentence-level multimodal citations. It automates data collection from Reddit for VQA, augments with a vast set of multimodal documents grounded by atomic facts, and employs human annotation to produce a reliable test set with fine-grained grounding criteria. The study shows multimodal retrieval-augmented generation improves informativeness and fluency over unimodal approaches but reveals weaker grounding for image-based documents, and demonstrates a knowledge extraction step can mitigate this bias. The findings highlight a persistent trade-off between informativeness and groundedness and point to future work in reducing contextual bias and extending attribution across additional modalities to enhance reliability of AI-generated, evidence-backed answers.

Abstract

Source attribution aims to enhance the reliability of AI-generated answers by including references for each statement, helping users validate the provided answers. However, existing work has primarily focused on text-only scenario and largely overlooked the role of multimodality. We introduce MAVIS, the first benchmark designed to evaluate multimodal source attribution systems that understand user intent behind visual questions, retrieve multimodal evidence, and generate long-form answers with citations. Our dataset comprises 157K visual QA instances, where each answer is annotated with fact-level citations referring to multimodal documents. We develop fine-grained automatic metrics along three dimensions of informativeness, groundedness, and fluency, and demonstrate their strong correlation with human judgments. Our key findings are threefold: (1) LVLMs with multimodal RAG generate more informative and fluent answers than unimodal RAG, but they exhibit weaker groundedness for image documents than for text documents, a gap amplified in multimodal settings. (2) Given the same multimodal documents, there is a trade-off between informativeness and groundedness across different prompting methods. (3) Our proposed method highlights mitigating contextual bias in interpreting image documents as a crucial direction for future research.

Figures (8)

• Figure 1: Example of the system's response in our MAVIS benchmark. Given a user question paired with an input image, the system must generate a long-form answer supported by sentence-level citations that reference multimodal documents. Highlighted texts can be verified using the corresponding colored documents.
• Figure 2: An illustration of the task formulation in our benchmark.
• Figure 3: Performance of Multi-RAG with GPT-4o using single-query ($N=1$) and multiple-query ($N=n$) retrieval. All baselines use Vanilla prompting. The $x$-axis indicates the total number of retrieved documents.
• Figure 4: Comparison of per-modality groundedness and document utilization between Unimodal-RAG (Text-RAG and Image-RAG), and Multimodal-RAG with and without knowledge extraction (KE). Each metric is averaged over the four LVLMs described in §\ref{['sec:models']}.
• Figure 5: Effect of adding text documents on image attention in GPT-4o. We fix the number of image documents at 5 in a single-query setting ($N=1$), and the x-axis represents the number of additional retrieved text documents. Documents are given in random order.