SuperCap: Multi-resolution Superpixel-based Image Captioning

TL;DR

SuperCap introduces a detector-free image captioning framework that uses SLIC superpixels as region units and extracts features with Vision-Language Models to capture object-like details while preserving global context through a multi-resolution design. The three-part pipeline—region extraction φ, region encoding Ψ via VLMs (BLIP or CLIP), and a Transformer-based captioning model Ω—enables end-to-end training on the COCO Karpathy split, achieving a CIDEr score of up to $136.9$ on the test set. Extensive ablations show a sweet spot around 10–15 superpixels, the benefit of including a global feature, and the superiority of BLIP-based encodings over CLIP and patch-based baselines. Overall, SuperCap provides a scalable, parameter-efficient approach that bridges detector-based and detector-free methods and remains competitive with SOTA detector-free models.

Abstract

It has been a longstanding goal within image captioning to move beyond a dependence on object detection. We investigate using superpixels coupled with Vision Language Models (VLMs) to bridge the gap between detector-based captioning architectures and those that solely pretrain on large datasets. Our novel superpixel approach ensures that the model receives object-like features whilst the use of VLMs provides our model with open set object understanding. Furthermore, we extend our architecture to make use of multi-resolution inputs, allowing our model to view images in different levels of detail, and use an attention mechanism to determine which parts are most relevant to the caption. We demonstrate our model's performance with multiple VLMs and through a range of ablations detailing the impact of different architectural choices. Our full model achieves a competitive CIDEr score of $136.9$ on the COCO Karpathy split.

Figures (4)

• Figure 1: Visual Representation of Different Region Extraction Techniques. Here we show the three different ways in which regions can be extracted from images, bounding boxes, patches, and superpixels.
• Figure 2: Architecture Diagram Our SuperCap model, like other image captioning models, has three main components: 1) the region extractor ($\varphi$), 2) the region encoder ($\Psi$), and 3) the captioning model ($\Omega$). The initial encoder takes in an image and uses BLIP to extract a global image feature. It simultaneously extracts superpixel regions for the image at different resolutions, which are in turn converted to BLIP region features. The secondary encoder consists of a stack of encoder transformers, one per resolution, which receive both the global feature and superpixel features for their resolution. Latent features produced by these encoders are then concatenated together and fed into a decoder transformer which autoregressively produces the final image caption.
• Figure 3: Test Set Performance. Images were randomly selected from the test set and captioned with the best performing model settings using beam width $5$. Best viewed in colour.
• Figure 4: Qualitative Performance. Example captions from the Karpathy test set. Best viewed in colour.