Real-Time Referring Expression Comprehension by Single-Stage Grounding Network

TL;DR

SSG introduces a real-time, end-to-end single-stage grounding model that localizes referents without region proposals by jointly encoding image and text, attending over local visual features, and directly regressing bounding boxes. It employs a multimodal interactor for attentive feature aggregation, a grounder with localization, guided attention, and attribute prediction modules, and a composite training objective that balances multiple losses. The guided-attention center-bias and attribute prediction components boost localization accuracy, while end-to-end training and a GPU-focused pipeline yield substantial speed-ups (≈40 referents per second). Across RefCOCO, RefCOCO+, RefCOCOg, and ReferItGame, SSG achieves competitive results and state-of-the-art performance on ReferItGame, demonstrating both effectiveness and practical efficiency for real-time grounding applications.

Abstract

In this paper, we propose a novel end-to-end model, namely Single-Stage Grounding network (SSG), to localize the referent given a referring expression within an image. Different from previous multi-stage models which rely on object proposals or detected regions, our proposed model aims to comprehend a referring expression through one single stage without resorting to region proposals as well as the subsequent region-wise feature extraction. Specifically, a multimodal interactor is proposed to summarize the local region features regarding the referring expression attentively. Subsequently, a grounder is proposed to localize the referring expression within the given image directly. For further improving the localization accuracy, a guided attention mechanism is proposed to enforce the grounder to focus on the central region of the referent. Moreover, by exploiting and predicting visual attribute information, the grounder can further distinguish the referent objects within an image and thereby improve the model performance. Experiments on RefCOCO, RefCOCO+, and RefCOCOg datasets demonstrate that our proposed SSG without relying on any region proposals can achieve comparable performance with other advanced models. Furthermore, our SSG outperforms the previous models and achieves the state-of-art performance on the ReferItGame dataset. More importantly, our SSG is time efficient and can ground a referring expression in a 416*416 image from the RefCOCO dataset in 25ms (40 referents per second) on average with a Nvidia Tesla P40, accomplishing more than 9* speedups over the existing multi-stage models.

Figures (10)

• Figure 1: A comparison between our SSG model and a traditional multi-stage method. By completely discarding the region proposal generation stage and directly predicting the bounding box for the referring expression, our SSG model runs faster by design.
• Figure 2: An overview of our proposed SSG model. The input image is encoded by a CNN to generate the local visual features representing different regions. An RNN encoder realized by a two-layer bidirectional LSTM (Bi-LSTM) is employed to process the referring expression sequentially and yield the textual feature. The multimodal interactor attentively exploits and summarizes the complicated relationships between the visual and textual features. In the referring expression grounder, the localization module relies on the joint context representations to yield the coordinates and the confidence score of the bounding box. Moreover, a novel guided attention mechanism by relating the attention weights to the referring region, enforces the visual attention to focus on the central region of the referent. Furthermore, the attribute prediction module is introduced to reproduce the attribute information contained in the referring expression. Please note that we only use the localization module to generate the bounding box for the referring expression during the inference stage.
• Figure 3: The illustration of our proposed guided attention loss. We formulate the guided attention process as a classification problem with the local region, where the central point falls into being labeled as 1 and the rest labeled as 0.
• Figure 4: Qualitative results of the referring expression comprehensions with the corresponding visual attention heat maps and top-5 predicted attributes. The red rectangles denote the ground-truth bounding boxes, while the yellow ones denote the predicted boxes by our SSG. The green dots indicate the center points of the ground-truth bounding boxes.
• Figure 5: Qualitative results of the referring expression comprehensions with the corresponding visual attention heat maps and top-5 predicted attributes. The red rectangles denote the ground-truth bounding boxes, while the yellow ones denote the predicted boxes by our SSG. The green dots indicate the center points of the ground-truth bounding boxes.