Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast Image Caption Generation with Position Alignment

Zheng-cong Fei

TL;DR

This work tackles the bottleneck of slow autoregressive image captioning by introducing FNIC, a two-stage non-autoregressive framework that leverages a position-aligned coarse word ordering to constrain generation. A light autoregressive GRU produces ordered words from image features, which then guide a non-autoregressive Transformer to generate final captions, with a latent-variable inference strategy improving robustness and fluency. Training jointly optimizes word-position alignment and sentence generation, while inference uses a non-deterministic approach to leverage the ordering distribution. Empirical results on COCO and Visual Genome show FNIC achieves competitive quality with substantial speedups (up to ~8x) over autoregressive models and outperforms general non-autoregressive baselines, demonstrating the value of explicit position information in NA captioning.

Abstract

Recent neural network models for image captioning usually employ an encoder-decoder architecture, where the decoder adopts a recursive sequence decoding way. However, such autoregressive decoding may result in sequential error accumulation and slow generation which limit the applications in practice. Non-autoregressive (NA) decoding has been proposed to cover these issues but suffers from language quality problem due to the indirect modeling of the target distribution. Towards that end, we propose an improved NA prediction framework to accelerate image captioning. Our decoding part consists of a position alignment to order the words that describe the content detected in the given image, and a fine non-autoregressive decoder to generate elegant descriptions. Furthermore, we introduce an inference strategy that regards position information as a latent variable to guide the further sentence generation. The Experimental results on public datasets show that our proposed model achieves better performance compared to general NA captioning models, while achieves comparable performance as autoregressive image captioning models with a significant speedup.

Fast Image Caption Generation with Position Alignment

TL;DR

This work tackles the bottleneck of slow autoregressive image captioning by introducing FNIC, a two-stage non-autoregressive framework that leverages a position-aligned coarse word ordering to constrain generation. A light autoregressive GRU produces ordered words from image features, which then guide a non-autoregressive Transformer to generate final captions, with a latent-variable inference strategy improving robustness and fluency. Training jointly optimizes word-position alignment and sentence generation, while inference uses a non-deterministic approach to leverage the ordering distribution. Empirical results on COCO and Visual Genome show FNIC achieves competitive quality with substantial speedups (up to ~8x) over autoregressive models and outperforms general non-autoregressive baselines, demonstrating the value of explicit position information in NA captioning.

Abstract

Recent neural network models for image captioning usually employ an encoder-decoder architecture, where the decoder adopts a recursive sequence decoding way. However, such autoregressive decoding may result in sequential error accumulation and slow generation which limit the applications in practice. Non-autoregressive (NA) decoding has been proposed to cover these issues but suffers from language quality problem due to the indirect modeling of the target distribution. Towards that end, we propose an improved NA prediction framework to accelerate image captioning. Our decoding part consists of a position alignment to order the words that describe the content detected in the given image, and a fine non-autoregressive decoder to generate elegant descriptions. Furthermore, we introduce an inference strategy that regards position information as a latent variable to guide the further sentence generation. The Experimental results on public datasets show that our proposed model achieves better performance compared to general NA captioning models, while achieves comparable performance as autoregressive image captioning models with a significant speedup.

Paper Structure

This paper contains 23 sections, 17 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Image Caption
  4. Non-autoregressive Decoder
  5. Method
  6. Image Encoder
  7. Hierarchical Decoder
  8. Position Aligment
  9. Fine Decoder
  10. Inference Procedure
  11. Training Procedure
  12. Experiments
  13. Dataset
  14. Baselines
  15. Implementation Details
  16. ...and 8 more sections

Figures (2)

  • Figure 1: The framework of our proposed NA captioning model. ① Different from general NA decoding models, our model adds a position alignment module between the image encoder module and sentence decoder module to explicitly construct object and position information. ② For a general NA model, the decoder inputs are the copied of source information, but for our model, the decoder inputs are ordered words with image information.
  • Figure 2: Image examples from Lin2014Microsoft with object regions and sentence generation results. The output sentences are generated by (1) Ground Truth (GT): One ground truth sentence, (2) Up-Down (3) GCN-LSTM (4) NAIC and (5) our proposed FNIC.