Dual-Stream Collaborative Transformer for Image Captioning

TL;DR

DSCT addresses contextual gaps in region-feature-based image captioning by fusing region and segmentation features through Pattern-Specific Mutual Attention Encoders and Dynamic Nomination Decoders. The two-stream transformer enables mutual refinement of features and dynamic, per-word path selection, circumventing semantic misalignment. Experiments on MS-COCO show state-of-the-art CIDEr scores for both single-model and ensemble setups, outperforming region-, grid-, and fusion-based rivals. This work demonstrates the value of segmentation features as complementary guidance and introduces a general dynamic fusion framework for captioning with multiple pattern-specific features.

Abstract

Current region feature-based image captioning methods have progressed rapidly and achieved remarkable performance. However, they are still prone to generating irrelevant descriptions due to the lack of contextual information and the over-reliance on generated partial descriptions for predicting the remaining words. In this paper, we propose a Dual-Stream Collaborative Transformer (DSCT) to address this issue by introducing the segmentation feature. The proposed DSCT consolidates and then fuses the region and segmentation features to guide the generation of caption sentences. It contains multiple Pattern-Specific Mutual Attention Encoders (PSMAEs) and Dynamic Nomination Decoders (DNDs). The PSMAE effectively highlights and consolidates the private information of two representations by querying each other. The DND dynamically searches for the most relevant learning blocks to the input textual representations and exploits the homogeneous features between the consolidated region and segmentation features to generate more accurate and descriptive caption sentences. To the best of our knowledge, this is the first study to explore how to fuse different pattern-specific features in a dynamic way to bypass their semantic inconsistencies and spatial misalignment issues for image captioning. The experimental results from popular benchmark datasets demonstrate that our DSCT outperforms the state-of-the-art image captioning models in the literature.

Figures (6)

• Figure 1: Baseline refers to the popular transformer-based image captioning model that uses only the region feature to generate caption sentences. (a) region features lack modeling of contextual information that easily lead to inaccurate captions (e.g., at a table). (b) region feature-based captioning models highly depend on the partially generated description for predicting the remaining words, which causes irrelevant descriptions (e.g., in the rain). The integration of region and segmentation features into our DSCT framework learns more effective visual and semantic representations and exploits the homogeneous features among them for more accurate and descriptive image captioning.
• Figure 2: The overall architecture of the proposed Dual-Stream Collaborative Transformer (DSCT). The proposed PSMAE consolidates both the region and segmentation representations by highlighting their private information and querying each other. Then, they are fused in a novel dynamic way by the proposed DND to guide the generation of caption sentences. By integrating the PSMAE and DND into a Dual-Stream Collaborative Transformer (DSCT) via a seamless formulation, the segmentation and region features are utilized more effectively for achieving more accurate and descriptive image captioning.
• Figure 3: The proposed Pattern-Specific Mutual Attention Encoder (PSMAE) consolidates each representation by highlighting its private information and querying the others to enhance the both representations for image captioning.
• Figure 4: The proposed Dynamic Nomination Decoder(DND) is a two-stream structure in which an proper stream is dynamically nominated to generate the next words by integrating a Dynamic Nomination Module. This design forces each stream to update its private information to learn more effective visual and semantic representations. Moreover, by stacking multiple DNDs, the semantic inconsistencies and spatial misalignment issues between region and segmentation features are bypassed, so that they are utilized more effectively for improving image captioning performance.
• Figure 5: Examples of image captioning results by baselines (such as DIFNet, RSTNet and our baseline transformer) and our proposed DSCT, along with the ground-truth sentences. Compared to other baselines, our proposed DSCT can identify more objects (red words) and accurately describe the relationships between objects (green words) in images, thereby generating more accurate and decriptive sentences.