Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Synthetic Multimodal Question Generation

Ian Wu, Sravan Jayanthi, Vijay Viswanathan, Simon Rosenberg, Sina Pakazad, Tongshuang Wu, Graham Neubig

TL;DR

The paper tackles the difficulty of evaluating multimodal retrieval-augmented QA (MMRAG) by introducing SMMQG, a synthetic data framework that generates question-and-answer pairs grounded in multimodal documents with fine-grained control over question styles and modalities. It leverages a retriever, an LLM, and an LMM in a closed loop to produce 1024 QA pairs from Wikipedia and demonstrates that the synthetic data yields actionable, style- and modality-specific insights into retriever and QA model performance. A human study shows SMMQG data quality is on par with crowdsourced MMQA, and concurrence analysis reveals strong alignment with MMQA in downstream evaluation. Collectively, SMMQG enables automatic, scalable, and tailored evaluation of MMRAG systems, supporting model selection and revealing weaknesses that fixed benchmarks might miss.

Abstract

Multimodal Retrieval Augmented Generation (MMRAG) is a powerful approach to question-answering over multimodal documents. A key challenge with evaluating MMRAG is the paucity of high-quality datasets matching the question styles and modalities of interest. In light of this, we propose SMMQG, a synthetic data generation framework. SMMQG leverages interplay between a retriever, large language model (LLM) and large multimodal model (LMM) to generate question and answer pairs directly from multimodal documents, with the questions conforming to specified styles and modalities. We use SMMQG to generate an MMRAG dataset of 1024 questions over Wikipedia documents and evaluate state-of-the-art models using it, revealing insights into model performance that are attainable only through style- and modality-specific evaluation data. Next, we measure the quality of data produced by SMMQG via a human study. We find that the quality of SMMQG-generated synthetic data is on par with the quality of the crowdsourced benchmark MMQA and that downstream evaluation results using both datasets strongly concur.

Synthetic Multimodal Question Generation

TL;DR

The paper tackles the difficulty of evaluating multimodal retrieval-augmented QA (MMRAG) by introducing SMMQG, a synthetic data framework that generates question-and-answer pairs grounded in multimodal documents with fine-grained control over question styles and modalities. It leverages a retriever, an LLM, and an LMM in a closed loop to produce 1024 QA pairs from Wikipedia and demonstrates that the synthetic data yields actionable, style- and modality-specific insights into retriever and QA model performance. A human study shows SMMQG data quality is on par with crowdsourced MMQA, and concurrence analysis reveals strong alignment with MMQA in downstream evaluation. Collectively, SMMQG enables automatic, scalable, and tailored evaluation of MMRAG systems, supporting model selection and revealing weaknesses that fixed benchmarks might miss.

Abstract

Multimodal Retrieval Augmented Generation (MMRAG) is a powerful approach to question-answering over multimodal documents. A key challenge with evaluating MMRAG is the paucity of high-quality datasets matching the question styles and modalities of interest. In light of this, we propose SMMQG, a synthetic data generation framework. SMMQG leverages interplay between a retriever, large language model (LLM) and large multimodal model (LMM) to generate question and answer pairs directly from multimodal documents, with the questions conforming to specified styles and modalities. We use SMMQG to generate an MMRAG dataset of 1024 questions over Wikipedia documents and evaluate state-of-the-art models using it, revealing insights into model performance that are attainable only through style- and modality-specific evaluation data. Next, we measure the quality of data produced by SMMQG via a human study. We find that the quality of SMMQG-generated synthetic data is on par with the quality of the crowdsourced benchmark MMQA and that downstream evaluation results using both datasets strongly concur.
Paper Structure (49 sections, 2 equations, 5 figures, 15 tables)

This paper contains 49 sections, 2 equations, 5 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Problem Setting
  3. Multimodal Sources
  4. Formulation
  5. Method
  6. SMMQG
  7. Multi-hop QA Generation
  8. Experiments
  9. Building a Synthetic Multimodal Wikipedia QA Dataset
  10. Retriever and QA Model Evaluation
  11. Assessing Data Quality
  12. MMQA
  13. Human Study
  14. Measuring Concurrence
  15. Related Work
  16. ...and 34 more sections

Figures (5)

  • Figure 1: An overview of SMMQG. Given user-provided question style and modality requirements, SMMQG selects question sources and produces questions and answers. The questions are grounded in the selected question sources, and adhere to the question and modality requirements.
  • Figure 2: SMMQG consists of five steps. (1) A seed source is sampled from the sources $S$. (2) An entity is extracted from the seed source. (3) Candidate sources are retrieved from $S$ using the entity from step 2 as the query. (4) The question generation model chooses the question sources from amongst the candidate sources, and uses these to produce the question and answer. (5) The model is asked to verify that the generated question adheres to the desired question style and modalities, and that the generated answer correctly answers the question.
  • Figure 3: Multi-hop Question Generation. Instead of directly producing a question as in standard SMMQG, we first generate two intermediate questions. The first question is about the extracted entity, and the second has the extracted entity as the answer. These are then combined by an LLM or LMM to produce the final, multi-hop question.
  • Figure 4: An example section from a Google Form shown to one of the crowdworkers. This section contains the question, answer, question style and source.
  • Figure 5: An example section from a Google Form shown to one of the crowdworkers. This section contains the response section.