Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Transformer based Multitask Learning for Image Captioning and Object Detection

Debolena Basak, P. K. Srijith, Maunendra Sankar Desarkar

TL;DR

This work introduces a novel multitask learning framework that combines image captioning and object detection into a joint model for jointly training both tasks by combining the losses obtained from image captioning and object detection networks.

Abstract

In several real-world scenarios like autonomous navigation and mobility, to obtain a better visual understanding of the surroundings, image captioning and object detection play a crucial role. This work introduces a novel multitask learning framework that combines image captioning and object detection into a joint model. We propose TICOD, Transformer-based Image Captioning and Object detection model for jointly training both tasks by combining the losses obtained from image captioning and object detection networks. By leveraging joint training, the model benefits from the complementary information shared between the two tasks, leading to improved performance for image captioning. Our approach utilizes a transformer-based architecture that enables end-to-end network integration for image captioning and object detection and performs both tasks jointly. We evaluate the effectiveness of our approach through comprehensive experiments on the MS-COCO dataset. Our model outperforms the baselines from image captioning literature by achieving a 3.65% improvement in BERTScore.

Transformer based Multitask Learning for Image Captioning and Object Detection

TL;DR

This work introduces a novel multitask learning framework that combines image captioning and object detection into a joint model for jointly training both tasks by combining the losses obtained from image captioning and object detection networks.

Abstract

In several real-world scenarios like autonomous navigation and mobility, to obtain a better visual understanding of the surroundings, image captioning and object detection play a crucial role. This work introduces a novel multitask learning framework that combines image captioning and object detection into a joint model. We propose TICOD, Transformer-based Image Captioning and Object detection model for jointly training both tasks by combining the losses obtained from image captioning and object detection networks. By leveraging joint training, the model benefits from the complementary information shared between the two tasks, leading to improved performance for image captioning. Our approach utilizes a transformer-based architecture that enables end-to-end network integration for image captioning and object detection and performs both tasks jointly. We evaluate the effectiveness of our approach through comprehensive experiments on the MS-COCO dataset. Our model outperforms the baselines from image captioning literature by achieving a 3.65% improvement in BERTScore.
Paper Structure (9 sections, 4 equations, 3 figures, 4 tables)

This paper contains 9 sections, 4 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Method
  4. Objective function
  5. Experimental Setup
  6. Results
  7. Comparison and Analysis
  8. Ablation Studies
  9. Conclusion

Figures (3)

  • Figure 1: A high-level framework of our proposed method. Our model TICOD has three major components, which we call as -- (a) the backbone network, (b) the object detection network, and (c) the caption network.
  • Figure 2: Architectural overview of the proposed Transformer-based Image Captioning and Object Detection (TICOD) model.
  • Figure 3: Examples of captions generated by $\mathcal{M}^2$-Transformer M2_transformer, PureT PureT, our model, and their corresponding ground-truths(GT) COCO_caps. The images also display the detected object categories and their scores as predicted by our proposed model.