Stacked Cross-modal Feature Consolidation Attention Networks for Image Captioning

TL;DR

The paper tackles image captioning by addressing the need for fine-grained, contextually grounded descriptions. It introduces stacked cross-modal feature consolidation (SCFC) attention networks that iteratively fuse visual region features with context-aware attributes and textual context, producing a discriminative cross-modal representation for caption generation via SCFC-LSTM. The approach is trained end-to-end and augmented with reinforcement learning, achieving state-of-the-art results on MSCOCO and Flickr30K and demonstrated through extensive ablations and analysis. Practically, SCFC enables richer, semantically informed captions while retaining end-to-end trainability, with potential for dynamic adjustment of reasoning steps during generation.

Abstract

Recently, the attention-enriched encoder-decoder framework has aroused great interest in image captioning due to its overwhelming progress. Many visual attention models directly leverage meaningful regions to generate image descriptions. However, seeking a direct transition from visual space to text is not enough to generate fine-grained captions. This paper exploits a feature-compounding approach to bring together high-level semantic concepts and visual information regarding the contextual environment fully end-to-end. Thus, we propose a stacked cross-modal feature consolidation (SCFC) attention network for image captioning in which we simultaneously consolidate cross-modal features through a novel compounding function in a multi-step reasoning fashion. Besides, we jointly employ spatial information and context-aware attributes (CAA) as the principal components in our proposed compounding function, where our CAA provides a concise context-sensitive semantic representation. To make better use of consolidated features potential, we further propose an SCFC-LSTM as the caption generator, which can leverage discriminative semantic information through the caption generation process. The experimental results indicate that our proposed SCFC can outperform various state-of-the-art image captioning benchmarks in terms of popular metrics on the MSCOCO and Flickr30K datasets.

Figures (7)

• Figure 1: The overview of the proposed SCFC for image captioning. Region proposals and attributes are extracted at the first step and then fed to the SCFC cell in each time step to consolidate cross-modal features through the caption generation process.
• Figure 2: The architecture of the coupled VD and AP in our approach. Given an image, the figure shows the process of detecting visual regions V and attributes A.
• Figure 3: The structure of our proposed SCFC-LSTM. Peephole connections and our consolidated input are shown with red and blue lines, respectively.
• Figure 4: Examples of attention weights' changes along with the generation of captions.
• Figure 5: Examples of caption results. The captions are generated by the Base model, SCFC, with one/two/three CFC layers in the yellow box to the solid green box, respectively.