Dual Attention Networks for Multimodal Reasoning and Matching

TL;DR

The paper tackles multimodal reasoning and matching by integrating visual and textual attention through memory-based mechanisms. It introduces two architectures, r-DAN for collaborative multimodal reasoning in VQA and m-DAN for learning a joint embedding for image-text matching, both built on multi-step attention with shared or separate memories. Key contributions include the joint memory update for r-DAN ($\mathbf{m}^{(k)} = \mathbf{m}^{(k-1)} + \mathbf{v}^{(k)} \odot \mathbf{u}^{(k)}$) and the dual-memory, cross-modal similarity objective for m-DAN ($S = \sum_{k=0}^{K} s^{(k)}$, with $s^{(k)} = \mathbf{v}^{(k)} \cdot \mathbf{u}^{(k)}$), trained via a max-margin ranking loss. Empirically, the DAN variants achieve state-of-the-art performance on Visual Question Answering and Flickr30K image-text matching, validating the effectiveness of coupled visual-textual attentions and memory-guided reasoning. The proposed framework is general and can be extended to tasks such as image captioning, visual grounding, and video question answering, highlighting its practical impact for vision-language understanding.

Abstract

We propose Dual Attention Networks (DANs) which jointly leverage visual and textual attention mechanisms to capture fine-grained interplay between vision and language. DANs attend to specific regions in images and words in text through multiple steps and gather essential information from both modalities. Based on this framework, we introduce two types of DANs for multimodal reasoning and matching, respectively. The reasoning model allows visual and textual attentions to steer each other during collaborative inference, which is useful for tasks such as Visual Question Answering (VQA). In addition, the matching model exploits the two attention mechanisms to estimate the similarity between images and sentences by focusing on their shared semantics. Our extensive experiments validate the effectiveness of DANs in combining vision and language, achieving the state-of-the-art performance on public benchmarks for VQA and image-text matching.

Figures (7)

• Figure 1: Overview of Dual Attention Networks (DANs) for multimodal reasoning and matching. The brightness of image regions and darkness of words indicate their attention weights predicted by DANs.
• Figure 2: Bidirectional LSTMs for text encoding.
• Figure 3: r-DAN in case of $K=2$.
• Figure 4: m-DAN in case of $K=2$.
• Figure 5: Qualitative results on the VQA dataset with attention visualization. For each example, the query image, question, and the answer by DAN are presented from top to bottom; the original image (question), the first and second attention maps are shown from left to right. The brightness of images and darkness of words represent their attention weights.