Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images

Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, Kuniko Saito

TL;DR

SlideVQA tackles the gap in document VQA by introducing a large multi-image dataset of slide decks that demands cross-slide and numerical reasoning. The authors propose M3D, an end-to-end Fusion-in-Decoder model that jointly performs evidence selection and QA, and can generate annotated arithmetic expressions to improve numerical reasoning. Experimental results show M3D outperforms baselines on joint evidence-question metrics but remains far from human performance, underscoring the challenge of multi-hop reasoning across images and accurate arithmetic. The dataset and model collectively contribute a scalable benchmark and a principled approach for building document understanding systems capable of handling complex, real-world slide decks.

Abstract

Visual question answering on document images that contain textual, visual, and layout information, called document VQA, has received much attention recently. Although many datasets have been proposed for developing document VQA systems, most of the existing datasets focus on understanding the content relationships within a single image and not across multiple images. In this study, we propose a new multi-image document VQA dataset, SlideVQA, containing 2.6k+ slide decks composed of 52k+ slide images and 14.5k questions about a slide deck. SlideVQA requires complex reasoning, including single-hop, multi-hop, and numerical reasoning, and also provides annotated arithmetic expressions of numerical answers for enhancing the ability of numerical reasoning. Moreover, we developed a new end-to-end document VQA model that treats evidence selection and question answering in a unified sequence-to-sequence format. Experiments on SlideVQA show that our model outperformed existing state-of-the-art QA models, but that it still has a large gap behind human performance. We believe that our dataset will facilitate research on document VQA.

SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images

TL;DR

SlideVQA tackles the gap in document VQA by introducing a large multi-image dataset of slide decks that demands cross-slide and numerical reasoning. The authors propose M3D, an end-to-end Fusion-in-Decoder model that jointly performs evidence selection and QA, and can generate annotated arithmetic expressions to improve numerical reasoning. Experimental results show M3D outperforms baselines on joint evidence-question metrics but remains far from human performance, underscoring the challenge of multi-hop reasoning across images and accurate arithmetic. The dataset and model collectively contribute a scalable benchmark and a principled approach for building document understanding systems capable of handling complex, real-world slide decks.

Abstract

Visual question answering on document images that contain textual, visual, and layout information, called document VQA, has received much attention recently. Although many datasets have been proposed for developing document VQA systems, most of the existing datasets focus on understanding the content relationships within a single image and not across multiple images. In this study, we propose a new multi-image document VQA dataset, SlideVQA, containing 2.6k+ slide decks composed of 52k+ slide images and 14.5k questions about a slide deck. SlideVQA requires complex reasoning, including single-hop, multi-hop, and numerical reasoning, and also provides annotated arithmetic expressions of numerical answers for enhancing the ability of numerical reasoning. Moreover, we developed a new end-to-end document VQA model that treats evidence selection and question answering in a unified sequence-to-sequence format. Experiments on SlideVQA show that our model outperformed existing state-of-the-art QA models, but that it still has a large gap behind human performance. We believe that our dataset will facilitate research on document VQA.
Paper Structure (45 sections, 3 equations, 7 figures, 5 tables)

This paper contains 45 sections, 3 equations, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Datasets for VQA on document images.
  4. Document VQA Models.
  5. Multi-modal question answering.
  6. VQA on videos or image sets.
  7. Slide images understanding.
  8. Reasoning over textual documents.
  9. The SlideVQA Task and Dataset
  10. Task Overview and Formulation
  11. Dataset Collection
  12. Slide decks collection.
  13. Bounding boxes and categories annotation.
  14. Single-hop QA creation.
  15. Multi-hop questions creation.
  16. ...and 30 more sections

Figures (7)

  • Figure 1: Examples from our SlideVQA dataset. Some questions can be answered through single-hop, multi-hop, and numerical reasoning. The colors of the words match the image borders with the same colors. ($\cdot$) of the right example in the answer denotes an annotated arithmetic expression to derive the final answer. The slide deck can be viewed at https://www.slideshare.net/mslgroup/mediainsights-evolving-sources-of-news-for-media.
  • Figure 2: Example of collected bounding boxes. Colored boxes and words were annotated by workers. The image can be viewed at https://www.slideshare.net/andrybrewok/big-data-analytics-a-social-network-approach.
  • Figure 3: Distribution of bounding box categories, reasoning types, numerical operations, and answer types in the test set.
  • Figure 4: Distribution of the first three words of the questions.
  • Figure 5: (a) Our encoder-decoder model architecture and (b) input representations. Given a question with a task prefix and a slide deck, the model outputs a corresponding answer/arithmetic-expression and evidence pages. The calculator outputs the final answer to calculate the generated arithmetic expression.
  • ...and 2 more figures