Self-paced Multi-grained Cross-modal Interaction Modeling for Referring Expression Comprehension

TL;DR

A Self-paced Multi-grained Cross-modal Interaction Modeling framework is proposed, which improves the language-to-vision localization ability through innovations in network structure and learning mechanism and significantly outperforms state-of-the-art methods on widely used datasets.

Abstract

As an important and challenging problem in vision-language tasks, referring expression comprehension (REC) generally requires a large amount of multi-grained information of visual and linguistic modalities to realize accurate reasoning. In addition, due to the diversity of visual scenes and the variation of linguistic expressions, some hard examples have much more abundant multi-grained information than others. How to aggregate multi-grained information from different modalities and extract abundant knowledge from hard examples is crucial in the REC task. To address aforementioned challenges, in this paper, we propose a Self-paced Multi-grained Cross-modal Interaction Modeling framework, which improves the language-to-vision localization ability through innovations in network structure and learning mechanism. Concretely, we design a transformer-based multi-grained cross-modal attention, which effectively utilizes the inherent multi-grained information in visual and linguistic encoders. Furthermore, considering the large variance of samples, we propose a self-paced sample informativeness learning to adaptively enhance the network learning for samples containing abundant multi-grained information. The proposed framework significantly outperforms state-of-the-art methods on widely used datasets, such as RefCOCO, RefCOCO+, RefCOCOg, and ReferItGame datasets, demonstrating the effectiveness of our method.

Figures (9)

• Figure 1: The diagram of the proposed Self-paced Multi-grained Cross-modal Interaction Modeling framework. CA$^{2}$M represents our proposed cross-modal alternate attention module. $v_1$, $v_2$, $v_3$ and $l_1$, $l_2$, $l_3$ represent the visual and linguistic features of the visual and linguistic encoders from shallow to deep layers, respectively.
• Figure 2: Comparison of the performance and inference speed on the val set of RefCOCO+. (a) and (b) denote comparisons without/with large-scale pre-training, respectively. Inference speed is tested on the 1080 Ti.
• Figure 3: Illustration of the proposed Self-paced Multi-grained Cross-modal Interaction Modeling framework. The whole framework consists of the visual encoder, linguistic encoder, multi-grained cross-modal attention, and self-paced sample informativeness learning. Concretely, visual and linguistic encoders first extract visual and linguistic features from different layers, respectively. In multi-grained cross-modal attention, visual and linguistic features obtained above are used to realize grouping and merging cross-modal interactions through the designed cross-modal alternate attention module (CA$^{2}$M). The [CLS] token, derived from linguistic features, aggregates visual and linguistic information during the linguistic feature extraction and multi-grained cross-modal attention, and is finally used for language-to-vision localization. A multi-layer perceptron converts the [CLS] token into the coordinates of the target box. During the training process, the output coordinates are used to realize self-paced sample informativeness learning. Best viewed in color.
• Figure 4: Illustration of the proposed cross-modal alternate attention module (CA$^{2}$M). $F_{v}$ and $F_{l}$ represent visual and linguistic features, respectively.
• Figure 5: Comparison with state-of-the-art methods resctransvg under different lengths of referring expression on RefCOCO, RefCOCO+, RefCOCOg and ReferItGame test sets.