Zero-Shot Visual Question Answering

TL;DR

Zero-shot Visual Question Answering tackles generalization by ensuring test questions/answers contain unseen words. The authors establish a zero-shot evaluation framework on Visual7W, create new splits, and evaluate a range of strategies, including pretrained word embeddings, stem-based sharing, test-time exemplar retrieval, and enhanced image representations. Extensive experiments show that these components improve zero-shot performance and, collectively, yield state-of-the-art results on standard VQA settings while exposing weaknesses in existing methods. The work underscores the value of auxiliary data and test-time retrieval for robust, vocab-general VQA and outlines promising directions for future research.

Abstract

Part of the appeal of Visual Question Answering (VQA) is its promise to answer new questions about previously unseen images. Most current methods demand training questions that illustrate every possible concept, and will therefore never achieve this capability, since the volume of required training data would be prohibitive. Answering general questions about images requires methods capable of Zero-Shot VQA, that is, methods able to answer questions beyond the scope of the training questions. We propose a new evaluation protocol for VQA methods which measures their ability to perform Zero-Shot VQA, and in doing so highlights significant practical deficiencies of current approaches, some of which are masked by the biases in current datasets. We propose and evaluate several strategies for achieving Zero-Shot VQA, including methods based on pretrained word embeddings, object classifiers with semantic embeddings, and test-time retrieval of example images. Our extensive experiments are intended to serve as baselines for Zero-Shot VQA, and they also achieve state-of-the-art performance in the standard VQA evaluation setting.

Figures (4)

• Figure 1: All test questions in our evaluation setting include words unseen in training examples, and used in the test question itself and/or in multiple-choice answers. This setting evaluates the capabilities of a VQA algorithm for generalization beyond its training examples. We demonstrate the benefit of additional sources of information, via pretrained intermediate representations (e.g. word embeddings and object detections) and during test-time (exemplar retrieval).
• Figure 2: Test questions from the proposed zero-shot test split of the Visual7W dataset. Each instance contains one or more unseen words (in blue boldface), i.e. not used in any training question or answer. Tick marks indicate the correct answers among the given multiple choices.
• Figure 3: Our neural network for VQA follows a straightforward architecture to evaluate the impact of various features representing the input question, image, and multiple-choice answers. We obtain their respective fixed-size vector representations $x^\mathsf{Q}$, $x^\mathsf{A}$, and $x^\mathsf{I}$ by concatenating bag-of-words of the different features. The three representations are passed through non-linear mappings (weights $\mathsf{w}$ followed by non-linearities not shown) and combined with multiplicative or order interactions. A final logistic regression over the combined features produces a score for each candidate answer.
• Figure 4: Individual impact of each proposed improvement on the standard (std.) and zero-shot (Z.S.) splits of the Visual7W dataset (average accuracy in %; note the different vertical scale on the bottom row). See discussion in Section \ref{['sect:experiments']}.