Document Visual Question Answering Challenge 2020

TL;DR

This paper introduces DocVQA, a large-scale benchmark for visual question answering on document images, featuring two tasks: extractive QA on single document images and retrieval-based QA over a collection. It provides two datasets: Task 1 with 50,000 questions on 12,767 scanned document images (with OCR transcriptions) and Task 2 with 14,362 images of a US Candidate Registration form template and 20 questions. Evaluation uses ANLS for Task 1 and MAP for Task 2, with multiple teams participating and reporting meaningful improvements over baselines; top methods include DQA (PingAn) for Task 1. The work advances document understanding in VQA, establishes a public challenge platform, and sets baselines for future research in document-centric visual QA and retrieval.

Abstract

This paper presents results of Document Visual Question Answering Challenge organized as part of "Text and Documents in the Deep Learning Era" workshop, in CVPR 2020. The challenge introduces a new problem - Visual Question Answering on document images. The challenge comprised two tasks. The first task concerns with asking questions on a single document image. On the other hand, the second task is set as a retrieval task where the question is posed over a collection of images. For the task 1 a new dataset is introduced comprising 50,000 questions-answer(s) pairs defined over 12,767 document images. For task 2 another dataset has been created comprising 20 questions over 14,362 document images which share the same document template.

Figures (3)

• Figure 1: (Left) A sample document image and questions defined on it from the dataset for task 1. (Right) A sample document image from dataset for task 2 and a sample question posed on the whole task 2 document collection.
• Figure 2: Performance of the submitted methods for different question types in task 1.
• Figure 3: Average precision of the submitted methods for each question in the test set.