Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WebQA: Multihop and Multimodal QA

Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, Yonatan Bisk

TL;DR

WebQA introduces a large-scale, multi-hop, multimodal open-domain QA benchmark that requires retrieving and aggregating information from both text and image sources, reasoning across multiple knowledge fragments, and generating fluent natural-language answers. It jointly evaluates source retrieval, cross-modal reasoning, and generation using a novel fluency- and accuracy-based metric, plus adversarial and hard-negative constructions to probe generalization. Baselines show that current vision-language models and even large language models struggle to fully utilize retrieved sources, highlighting the need for unified multimodal representations and scalable retrieval-then-reasoning pipelines. The dataset, with restricted and full retrieval settings, serves as a platform to push toward digital assistants capable of open-world visual and textual knowledge integration. Finally, WebQA provides guidance on evaluation and data collection practices to foster progress in open-domain, multimodal reasoning research.

Abstract

Scaling Visual Question Answering (VQA) to the open-domain and multi-hop nature of web searches, requires fundamental advances in visual representation learning, knowledge aggregation, and language generation. In this work, we introduce WebQA, a challenging new benchmark that proves difficult for large-scale state-of-the-art models which lack language groundable visual representations for novel objects and the ability to reason, yet trivial for humans. WebQA mirrors the way humans use the web: 1) Ask a question, 2) Choose sources to aggregate, and 3) Produce a fluent language response. This is the behavior we should be expecting from IoT devices and digital assistants. Existing work prefers to assume that a model can either reason about knowledge in images or in text. WebQA includes a secondary text-only QA task to ensure improved visual performance does not come at the cost of language understanding. Our challenge for the community is to create unified multimodal reasoning models that answer questions regardless of the source modality, moving us closer to digital assistants that not only query language knowledge, but also the richer visual online world.

WebQA: Multihop and Multimodal QA

TL;DR

WebQA introduces a large-scale, multi-hop, multimodal open-domain QA benchmark that requires retrieving and aggregating information from both text and image sources, reasoning across multiple knowledge fragments, and generating fluent natural-language answers. It jointly evaluates source retrieval, cross-modal reasoning, and generation using a novel fluency- and accuracy-based metric, plus adversarial and hard-negative constructions to probe generalization. Baselines show that current vision-language models and even large language models struggle to fully utilize retrieved sources, highlighting the need for unified multimodal representations and scalable retrieval-then-reasoning pipelines. The dataset, with restricted and full retrieval settings, serves as a platform to push toward digital assistants capable of open-world visual and textual knowledge integration. Finally, WebQA provides guidance on evaluation and data collection practices to foster progress in open-domain, multimodal reasoning research.

Abstract

Scaling Visual Question Answering (VQA) to the open-domain and multi-hop nature of web searches, requires fundamental advances in visual representation learning, knowledge aggregation, and language generation. In this work, we introduce WebQA, a challenging new benchmark that proves difficult for large-scale state-of-the-art models which lack language groundable visual representations for novel objects and the ability to reason, yet trivial for humans. WebQA mirrors the way humans use the web: 1) Ask a question, 2) Choose sources to aggregate, and 3) Produce a fluent language response. This is the behavior we should be expecting from IoT devices and digital assistants. Existing work prefers to assume that a model can either reason about knowledge in images or in text. WebQA includes a secondary text-only QA task to ensure improved visual performance does not come at the cost of language understanding. Our challenge for the community is to create unified multimodal reasoning models that answer questions regardless of the source modality, moving us closer to digital assistants that not only query language knowledge, but also the richer visual online world.

Paper Structure

This paper contains 96 sections, 3 equations, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Task Formulation
  4. WebQA
  5. Answers from Images
  6. Hard Negative Mining
  7. Categorization
  8. Adversarial splits
  9. Answers From Text
  10. Hard Negative Mining
  11. Quality Control
  12. Dataset Statistics
  13. Multi-hop
  14. Topics
  15. Metrics
  16. ...and 81 more sections

Figures (4)

  • Figure 1: Example WebQA dataset pipeline in which the question requires finding and reasoning about two relevant sources and discarding distractors to produce the correct natural language answer.
  • Figure 2: Image question prefixes (see Appendix B).
  • Figure 3: Samples of common topics in the image-based (left) and text-based (right) folds of the data.
  • Figure 4: Few-Shot GPT-3 prompts.