Multimodal Residual Learning for Visual QA

TL;DR

The paper tackles visual question answering by learning joint vision-language representations through Multimodal Residual Networks (MRN), which extend deep residual learning with a joint residual function based on element-wise multiplication. MRN stacks multiple learning blocks with identity shortcuts for the visual stream, enabling deeper multimodal representations and implicit attention without dedicated attention parameters. The authors demonstrate state-of-the-art results on both Open-Ended and Multiple-Choice VQA on the VQA dataset, and introduce a back-propagation–based visualization to reveal spatial attention effects despite collapsed visual features. The work highlights the potential of residual-based multimodal architectures and proposes a practical, interpretable visualization approach to analyze where the model attends in images during reasoning.

Abstract

Deep neural networks continue to advance the state-of-the-art of image recognition tasks with various methods. However, applications of these methods to multimodality remain limited. We present Multimodal Residual Networks (MRN) for the multimodal residual learning of visual question-answering, which extends the idea of the deep residual learning. Unlike the deep residual learning, MRN effectively learns the joint representation from vision and language information. The main idea is to use element-wise multiplication for the joint residual mappings exploiting the residual learning of the attentional models in recent studies. Various alternative models introduced by multimodality are explored based on our study. We achieve the state-of-the-art results on the Visual QA dataset for both Open-Ended and Multiple-Choice tasks. Moreover, we introduce a novel method to visualize the attention effect of the joint representations for each learning block using back-propagation algorithm, even though the visual features are collapsed without spatial information.

Figures (7)

• Figure 1: Inference flow of Multimodal Residual Networks (MRN). Using our visualization method, the attention effects are shown as a sequence of three images. More examples are shown in Figure \ref{['fig:examples']}.
• Figure 2: A schematic diagram of Multimodal Residual Networks with three-block layers.
• Figure 3: Alternative models are explored to justify our proposed model. The base model (a) has a shortcut for a question vector as SAN does Yang2015, and the joint residual function takes the form of the Deep Q+I model's joint function Lu2015. (b) extra embedding for visual modality. (c) extra embeddings for both modalities. (d) identity mappings for shortcuts. In the first learning block, use a linear mapping for matching a dimension with the joint dimension. (e) two shortcuts for both modalities. For simplicity, the linear mapping of visual shortcut only appears in the first learning block. Notice that (d) and (e) are compared to (b) after the model selection of (b) among (a)-(c) on test-dev results. Eventually, we chose (b) as the best performance and relative simplicity.
• Figure 4: Examples for visualization of the three-block layered MRN. The original images are shown in the first of each group. The next three images show the input gradients of the attention effect for each learning block as described in Section \ref{['subsec:visualization']}. The gradients of color channels for each pixel are summed up after taking absolute values of these gradients. Then, these summed absolute values which are greater than the summation of the mean and the standard deviation of these values are visualized as the attention effect (bright color) on the images. The answers (blue) are predicted by MRN.
• Figure 5: More examples of Figure 4 in Section 5.2.