Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tri-FusionNet: Enhancing Image Description Generation with Transformer-based Fusion Network and Dual Attention Mechanism

Lakshita Agarwal, Bindu Verma

TL;DR

Tri-FusionNet tackles image caption generation by fusing Vision Transformer-based encoding with a dual-attention mechanism, a RoBERTa decoder, and a CLIP integration to align vision and language representations. The architecture leverages ViT for local/global feature extraction, RoBERTa for robust language modeling, and CLIP for cross-modal fusion via contrastive learning, resulting in high-quality, context-rich captions. Empirical results on MSCOCO, Flickr30k, and Flickr8k show state-of-the-art-like gains across BLEU, CIDEr, METEOR, and ROUGE-L metrics, validating the effectiveness of the fusion strategy. The approach offers practical implications for accessibility and multimodal AI systems, while highlighting deployment and bias considerations for real-world use.

Abstract

Image description generation is essential for accessibility and AI understanding of visual content. Recent advancements in deep learning have significantly improved natural language processing and computer vision. In this work, we propose Tri-FusionNet, a novel image description generation model that integrates transformer modules: a Vision Transformer (ViT) encoder module with dual-attention mechanism, a Robustly Optimized BERT Approach (RoBERTa) decoder module, and a Contrastive Language-Image Pre-Training (CLIP) integrating module. The ViT encoder, enhanced with dual attention, focuses on relevant spatial regions and linguistic context, improving image feature extraction. The RoBERTa decoder is employed to generate precise textual descriptions. CLIP's integrating module aligns visual and textual data through contrastive learning, ensuring effective combination of both modalities. This fusion of ViT, RoBERTa, and CLIP, along with dual attention, enables the model to produce more accurate, contextually rich, and flexible descriptions. The proposed framework demonstrated competitive performance on the Flickr30k and Flickr8k datasets, with BLEU scores ranging from 0.767 to 0.456 and 0.784 to 0.479, CIDEr scores of 1.679 and 1.483, METEOR scores of 0.478 and 0.358, and ROUGE-L scores of 0.567 and 0.789, respectively. On MS-COCO, the framework obtained BLEU scores of 0.893 (B-1), 0.821 (B-2), 0.794 (B-3), and 0.725 (B-4). The results demonstrate the effectiveness of Tri-FusionNet in generating high-quality image descriptions.

Tri-FusionNet: Enhancing Image Description Generation with Transformer-based Fusion Network and Dual Attention Mechanism

TL;DR

Tri-FusionNet tackles image caption generation by fusing Vision Transformer-based encoding with a dual-attention mechanism, a RoBERTa decoder, and a CLIP integration to align vision and language representations. The architecture leverages ViT for local/global feature extraction, RoBERTa for robust language modeling, and CLIP for cross-modal fusion via contrastive learning, resulting in high-quality, context-rich captions. Empirical results on MSCOCO, Flickr30k, and Flickr8k show state-of-the-art-like gains across BLEU, CIDEr, METEOR, and ROUGE-L metrics, validating the effectiveness of the fusion strategy. The approach offers practical implications for accessibility and multimodal AI systems, while highlighting deployment and bias considerations for real-world use.

Abstract

Image description generation is essential for accessibility and AI understanding of visual content. Recent advancements in deep learning have significantly improved natural language processing and computer vision. In this work, we propose Tri-FusionNet, a novel image description generation model that integrates transformer modules: a Vision Transformer (ViT) encoder module with dual-attention mechanism, a Robustly Optimized BERT Approach (RoBERTa) decoder module, and a Contrastive Language-Image Pre-Training (CLIP) integrating module. The ViT encoder, enhanced with dual attention, focuses on relevant spatial regions and linguistic context, improving image feature extraction. The RoBERTa decoder is employed to generate precise textual descriptions. CLIP's integrating module aligns visual and textual data through contrastive learning, ensuring effective combination of both modalities. This fusion of ViT, RoBERTa, and CLIP, along with dual attention, enables the model to produce more accurate, contextually rich, and flexible descriptions. The proposed framework demonstrated competitive performance on the Flickr30k and Flickr8k datasets, with BLEU scores ranging from 0.767 to 0.456 and 0.784 to 0.479, CIDEr scores of 1.679 and 1.483, METEOR scores of 0.478 and 0.358, and ROUGE-L scores of 0.567 and 0.789, respectively. On MS-COCO, the framework obtained BLEU scores of 0.893 (B-1), 0.821 (B-2), 0.794 (B-3), and 0.725 (B-4). The results demonstrate the effectiveness of Tri-FusionNet in generating high-quality image descriptions.

Paper Structure

This paper contains 13 sections, 7 equations, 3 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Framework
  4. Overview of the Proposed Architecture:
  5. Data Pre-processing
  6. Vision Transformer Encoder Module with Dual Attention Mechanism
  7. RoBERTa Decoder
  8. CLIP Integration Module
  9. Experimental Analysis
  10. Ablation Study:
  11. Results and Analysis:
  12. Comparison with Other State-of-the-art Methods:
  13. Conclusion and Future Direction

Figures (3)

  • Figure 1: Structural representation of Tri-FusionNet framework: The architecture consists of three phases: firstly, high-level visual features are first extracted from pre-processed images using a Vision transformer encoder with dual-attention mechanism; next, words from the input caption file are tokenized by a RoBERTa decoder; and last, the combined data is fed into a CLIP-integrating module to create image descriptions using dense network layers.
  • Figure 2: Example of obtained heat maps based on dual-attention mechanism.
  • Figure 3: Graphical representation of the results obtained from the datasets for the proposed Tri-FusionNet framework.