Image Captioning via Compact Bidirectional Architecture

TL;DR

A Compact Bidirectional Transformer model for image captioning that can leverage bidirectional context implicitly and explicitly while the decoder can be executed parallelly is introduced and extensive ablation studies on MSCOCO benchmark find that the compact bidirectional architecture and the sentence-level ensemble play more important roles than the explicit interaction mechanism.

Abstract

Most current image captioning models typically generate captions from left-to-right. This unidirectional property makes them can only leverage past context but not future context. Though refinement-based models can exploit both past and future context by generating a new caption in the second stage based on pre-retrieved or pre-generated captions in the first stage, the decoder of these models generally consists of two networks~(i.e. a retriever or captioner in the first stage and a captioner in the second stage), which can only be executed sequentially. In this paper, we introduce a Compact Bidirectional Transformer model for image captioning that can leverage bidirectional context implicitly and explicitly while the decoder can be executed parallelly. Specifically, it is implemented by tightly coupling left-to-right(L2R) and right-to-left(R2L) flows into a single compact model to serve as a regularization for implicitly exploiting bidirectional context and optionally allowing explicit interaction of the bidirectional flows, while the final caption is chosen from either L2R or R2L flow in a sentence-level ensemble manner. We conduct extensive ablation studies on MSCOCO benchmark and find that the compact bidirectional architecture and the sentence-level ensemble play more important roles than the explicit interaction mechanism. By combining with word-level ensemble seamlessly, the effect of sentence-level ensemble is further enlarged. We further extend the conventional one-flow self-critical training to the two-flows version under this architecture and achieve new state-of-the-art results in comparison with non-vision-language-pretraining models. Finally, we verify the generality of this compact bidirectional architecture by extending it to LSTM backbone. Source code is available at https://github.com/YuanEZhou/cbtic.

Figures (5)

• Figure 1: A conceptual overview of (a) Uni-directional generation,including left-to-right(L2R) and right-to-left(R2L), (b) Refinement-based generation, (c) Sentence-Level Ensemble and (d) our proposed Compact Bidirectional Transformer for Image Captioning (CBTrans), which combines the advantage of (abc), where L2R and R2L 'flows' share an unified network and interaction between the two 'flows' is optionally allowed inside.
• Figure 2: Illustration of Compact Bidirectional Transformer for Image Captioning (CBTrans). CBTrans model composes of an encoder (in the left) and a decoder (in the middle). An intuitional illustration of the extension of the standard Scaled Dot-Product Attention module is shown on the right, enabling explicit bidirectional context interaction (see Eq. (5)–(6)). Notice that the Residual Connections, Layer Normalization are omitted. The subscript of head $i$ in Q/K/V/H is omitted for brevity. Right-to-right (R2L) flow is marked in red. Best view in color.
• Figure 3: Illustration of Compact Bidirectional LSTM model (CBLSTM) for Image Captioning. Best view in color.
• Figure 4: Examples of captions generated by our CBTrans model, conventional unidirectional Transformer model and human-annotated ground truth. Some bad words are marked in red.
• Figure 5: Examples of captions generated by our CBLSTM model, conventional unidirectional Up-Down model and human-annotated ground truth.