Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Shifted Window Fourier Transform And Retention For Image Captioning

Jia Cheng Hu, Roberto Cavicchioli, Alessandro Capotondi

TL;DR

This work targets efficient image captioning for resource-constrained settings by introducing SwiFTeR, a lightweight architecture that combines a Shifted Window Fourier Transform visual backbone with a Retention-based fusion decoder. With about 20M parameters and ~3.1 GFLOPs per forward pass, SwiFTeR delivers high throughput (up to ~400 captions/s) and favorable scalability and memory properties, though current caption quality (CIDEr-D around 110) lags behind transformer-based models due to a weaker training setup. The paper provides thorough ablations, compares against state-of-the-art, and analyzes efficiency, power, and memory to demonstrate the practical benefits of the architecture for low-resource deployment. It points to promising future improvements through stronger backbones and larger-scale pre-training of the fusion model to close the performance gap while preserving the efficiency gains.

Abstract

Image Captioning is an important Language and Vision task that finds application in a variety of contexts, ranging from healthcare to autonomous vehicles. As many real-world applications rely on devices with limited resources, much effort in the field was put into the development of lighter and faster models. However, much of the current optimizations focus on the Transformer architecture in contrast to the existence of more efficient methods. In this work, we introduce SwiFTeR, an architecture almost entirely based on Fourier Transform and Retention, to tackle the main efficiency bottlenecks of current light image captioning models, being the visual backbone's onerosity, and the decoder's quadratic cost. SwiFTeR is made of only 20M parameters, and requires 3.1 GFLOPs for a single forward pass. Additionally, it showcases superior scalability to the caption length and its small memory requirements enable more images to be processed in parallel, compared to the traditional transformer-based architectures. For instance, it can generate 400 captions in one second. Although, for the time being, the caption quality is lower (110.2 CIDEr-D), most of the decrease is not attributed to the architecture but rather an incomplete training practice which currently leaves much room for improvements. Overall, SwiFTeR points toward a promising direction to new efficient architectural design. The implementation code will be released in the future.

Shifted Window Fourier Transform And Retention For Image Captioning

TL;DR

This work targets efficient image captioning for resource-constrained settings by introducing SwiFTeR, a lightweight architecture that combines a Shifted Window Fourier Transform visual backbone with a Retention-based fusion decoder. With about 20M parameters and ~3.1 GFLOPs per forward pass, SwiFTeR delivers high throughput (up to ~400 captions/s) and favorable scalability and memory properties, though current caption quality (CIDEr-D around 110) lags behind transformer-based models due to a weaker training setup. The paper provides thorough ablations, compares against state-of-the-art, and analyzes efficiency, power, and memory to demonstrate the practical benefits of the architecture for low-resource deployment. It points to promising future improvements through stronger backbones and larger-scale pre-training of the fusion model to close the performance gap while preserving the efficiency gains.

Abstract

Image Captioning is an important Language and Vision task that finds application in a variety of contexts, ranging from healthcare to autonomous vehicles. As many real-world applications rely on devices with limited resources, much effort in the field was put into the development of lighter and faster models. However, much of the current optimizations focus on the Transformer architecture in contrast to the existence of more efficient methods. In this work, we introduce SwiFTeR, an architecture almost entirely based on Fourier Transform and Retention, to tackle the main efficiency bottlenecks of current light image captioning models, being the visual backbone's onerosity, and the decoder's quadratic cost. SwiFTeR is made of only 20M parameters, and requires 3.1 GFLOPs for a single forward pass. Additionally, it showcases superior scalability to the caption length and its small memory requirements enable more images to be processed in parallel, compared to the traditional transformer-based architectures. For instance, it can generate 400 captions in one second. Although, for the time being, the caption quality is lower (110.2 CIDEr-D), most of the decrease is not attributed to the architecture but rather an incomplete training practice which currently leaves much room for improvements. Overall, SwiFTeR points toward a promising direction to new efficient architectural design. The implementation code will be released in the future.
Paper Structure (14 sections, 10 equations, 3 figures, 2 tables)

This paper contains 14 sections, 10 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. SwiFTeR Design
  4. Shifted Window Fourier Transform
  5. Retention Layer
  6. SwiFTeR
  7. Experimental Setup
  8. Dataset and Training
  9. Results
  10. Ablation Study
  11. Comparison With State-of-the-Art
  12. Efficiency
  13. Power Consumption and Memory
  14. Conclusion and Future Works

Figures (3)

  • Figure 1: Computational and parameters profiling of the baseline Image Captioning model consisting of three layers of Transformer vaswani2017attention and Swin-Transformer-S liu2021swin backbone. GPU Time is calculated as the average cost to generate captions of images from the COCO lin2014microsoft test set.
  • Figure 2: The SwiFTeR architecture. Normalization layers and skip connections are omitted. FT="Fourier Transform", MHA="MultiHead-Attention", FF/MLP="FeedForward".
  • Figure 3: Latency of SwiFTeR and baseline models with different length and batch size configurations. Unless differently stated, the length is set to 32, and the batch size is set to 32. The GPU consists of an NVIDIA GeForce RTX 4090 24GB. Each configuration is accommodated into one batch. Intel(R) Core i9-13900K is adopted as CPU.