Towards a Visual Turing Challenge

TL;DR

The paper advocates a Visual Turing Test for open-domain question answering on real-world images to advance holistic vision-language systems. It analyzes challenges in scalability, ambiguity, and the integration of common-sense knowledge, arguing that social consensus should guide evaluation under inherent human disagreement. DAQUAR is introduced as a challenging indoor-image QA dataset to illustrate these issues, highlighting linguistic richness and the need for non-visual cues. The authors propose metrics such as WUPS and new approaches like Interpretation and Consensus metrics, and outline experimental scenarios to assess generalization and open-domain reasoning in holistic architectures.

Abstract

As language and visual understanding by machines progresses rapidly, we are observing an increasing interest in holistic architectures that tightly interlink both modalities in a joint learning and inference process. This trend has allowed the community to progress towards more challenging and open tasks and refueled the hope at achieving the old AI dream of building machines that could pass a turing test in open domains. In order to steadily make progress towards this goal, we realize that quantifying performance becomes increasingly difficult. Therefore we ask how we can precisely define such challenges and how we can evaluate different algorithms on this open tasks? In this paper, we summarize and discuss such challenges as well as try to give answers where appropriate options are available in the literature. We exemplify some of the solutions on a recently presented dataset of question-answering task based on real-world indoor images that establishes a visual turing challenge. Finally, we argue despite the success of unique ground-truth annotation, we likely have to step away from carefully curated dataset and rather rely on 'social consensus' as the main driving force to create suitable benchmarks. Providing coverage in this inherently ambiguous output space is an emerging challenge that we face in order to make quantifiable progress in this area.