Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Image Captioning: Transforming Objects into Words

Simao Herdade, Armin Kappeler, Kofi Boakye, Joao Soares

TL;DR

The paper tackles image captioning by integrating spatial relationships between detected objects into a Transformer framework. It introduces the Object Relation Transformer, which augments attention with geometry-based weights derived from bounding-box coordinates and sizes, enabling explicit relational reasoning. Empirical results on MS-COCO 2014 show consistent improvements across CIDEr-D, SPICE, BLEU, METEOR, and ROUGE-L, with statistical significance in several metrics and notable gains in object counting and relational understanding. The work highlights the value of incorporating geometric information in visual-language models and points to future work extending geometry into decoder cross-attention for further gains.

Abstract

Image captioning models typically follow an encoder-decoder architecture which uses abstract image feature vectors as input to the encoder. One of the most successful algorithms uses feature vectors extracted from the region proposals obtained from an object detector. In this work we introduce the Object Relation Transformer, that builds upon this approach by explicitly incorporating information about the spatial relationship between input detected objects through geometric attention. Quantitative and qualitative results demonstrate the importance of such geometric attention for image captioning, leading to improvements on all common captioning metrics on the MS-COCO dataset.

Image Captioning: Transforming Objects into Words

TL;DR

The paper tackles image captioning by integrating spatial relationships between detected objects into a Transformer framework. It introduces the Object Relation Transformer, which augments attention with geometry-based weights derived from bounding-box coordinates and sizes, enabling explicit relational reasoning. Empirical results on MS-COCO 2014 show consistent improvements across CIDEr-D, SPICE, BLEU, METEOR, and ROUGE-L, with statistical significance in several metrics and notable gains in object counting and relational understanding. The work highlights the value of incorporating geometric information in visual-language models and points to future work extending geometry into decoder cross-attention for further gains.

Abstract

Image captioning models typically follow an encoder-decoder architecture which uses abstract image feature vectors as input to the encoder. One of the most successful algorithms uses feature vectors extracted from the region proposals obtained from an object detector. In this work we introduce the Object Relation Transformer, that builds upon this approach by explicitly incorporating information about the spatial relationship between input detected objects through geometric attention. Quantitative and qualitative results demonstrate the importance of such geometric attention for image captioning, leading to improvements on all common captioning metrics on the MS-COCO dataset.

Paper Structure

This paper contains 15 sections, 9 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Approach
  4. Object Detection
  5. Standard Transformer Model
  6. Object Relation Transformer
  7. Implementation Details
  8. Experimental Evaluation
  9. Dataset and Metrics
  10. Comparative Analysis
  11. Positional Encoding
  12. Ablation Study
  13. Geometric Improvement
  14. Qualitative Analysis
  15. Conclusion

Figures (2)

  • Figure 1: A visualization of self-attention in our proposed Object Relation Transformer. The transparency of the detected object and its bounding box is proportional to the attention weight with respect to the chair outlined in red. Our model strongly correlates this chair with the companion chair to the left, the beach beneath them, and the umbrella above them, relationships displayed in the generated caption.
  • Figure 2: Overview of Object Relation Transformer architecture. The Bounding Box Relational Encoding diagram describes the changes made to the Transformer architecture