Scene Text Visual Question Answering
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusiñol, Ernest Valveny, C. V. Jawahar, Dimosthenis Karatzas
TL;DR
This work introduces ST-VQA, a dataset and framework that require reading scene text to answer questions, addressing a gap in traditional VQA where text is neglected. It provides images from diverse sources, three increasingly challenging tasks with per-image and global dictionaries, and a new ANLS metric to jointly evaluate reasoning and OCR accuracy. Baseline experiments show that incorporating textual information improves VQA performance and that purely visual models underperform compared to text-aware approaches, highlighting the need for end-to-end reading systems. The paper also outlines future directions toward generative, multi-word answers and better integration of language priors in scene-text VQA, with an online evaluation service for public benchmarking.</
Abstract
Current visual question answering datasets do not consider the rich semantic information conveyed by text within an image. In this work, we present a new dataset, ST-VQA, that aims to highlight the importance of exploiting high-level semantic information present in images as textual cues in the VQA process. We use this dataset to define a series of tasks of increasing difficulty for which reading the scene text in the context provided by the visual information is necessary to reason and generate an appropriate answer. We propose a new evaluation metric for these tasks to account both for reasoning errors as well as shortcomings of the text recognition module. In addition we put forward a series of baseline methods, which provide further insight to the newly released dataset, and set the scene for further research.