ViPCap: Retrieval Text-Based Visual Prompts for Lightweight Image Captioning

TL;DR

ViPCap tackles the inefficiency of retrieval-based captioning by converting retrieved text into visual prompts that enrich local image representations. It introduces the ViP module, which maps retrieved text into the CLIP space and samples semantic features from a learnable Gaussian distribution, followed by a patch-level retrieval and a minimal Feature Fusion Network to fuse visual and semantic information. The approach yields strong performance among lightweight captioning methods on COCO, Flickr30k, and NoCaps, while maintaining a small parameter footprint and offering plug-and-play compatibility across models and prompts. This work enables efficient, robust multimodal captioning suitable for real-world deployment without retraining large encoders, and the code is publicly available.

Abstract

Recent lightweight image captioning models using retrieved data mainly focus on text prompts. However, previous works only utilize the retrieved text as text prompts, and the visual information relies only on the CLIP visual embedding. Because of this issue, there is a limitation that the image descriptions inherent in the prompt are not sufficiently reflected in the visual embedding space. To tackle this issue, we propose ViPCap, a novel retrieval text-based visual prompt for lightweight image captioning. ViPCap leverages the retrieved text with image information as visual prompts to enhance the ability of the model to capture relevant visual information. By mapping text prompts into the CLIP space and generating multiple randomized Gaussian distributions, our method leverages sampling to explore randomly augmented distributions and effectively retrieves the semantic features that contain image information. These retrieved features are integrated into the image and designated as the visual prompt, leading to performance improvements on the datasets such as COCO, Flickr30k, and NoCaps. Experimental results demonstrate that ViPCap significantly outperforms prior lightweight captioning models in efficiency and effectiveness, demonstrating the potential for a plug-and-play solution. The source code is available at https://github.com/taewhankim/VIPCAP.

Figures (4)

• Figure 1: (a) $\text{ViPCap}$ shows the best efficiency among the lightweight captioning models. (b) EVCap and MAPL require large models such as EVA-CLIP evaclip, Vicuna vicuna2023, and GPT-J gpt-j. With under 0.3B total parameters, $\text{ViPCap}$ achieves competitive performance despite its small size.
• Figure 2: SmallCap smallcap fails to accurately represent visual information, such as a local object, in the ground truth (GT) or retrieval text (Retrieval). In contrast, our $\text{ViPCap}$ accurately captures visual information found in GT or retrieval text.
• Figure 3: $\text{ViPCap}$ leverages the CLIP text encoder to extract retrieved text features for visual prompts generation. The $\text{ViP}$ module performs $M$ sampling iterations from the text embedding distribution to extract semantic features $G$ closely aligned with image patch features $V$. Then, in the patch retrieval module, we retrieve semantic vectors from $G$ that are highly correlated with image patch features $V$. The retrieved semantic features $R$ are fused with image features $V$ within the Feature Fusion Network, and the resulting output is set as the visual prompt $Z$. Finally, the refined visual feature $V'$ via summation with the visual prompt is fed to the decoder through the cross-attention layer.
• Figure 4: Calculating similarity between input feature $V$ and the semantic features $G$. $V$ retrieves essential semantic representations from $G$ and combines these retrieved semantic features $R$ through the fusion network. The fusion network generates a visual prompt $Z$ by integrating image features $V$. After that, the visual prompt and image features are combined to create refined visual features $V'$.