Group-based Distinctive Image Captioning with Memory Difference Encoding and Attention

TL;DR

This work tackles the problem of generating distinctive image captions that can differentiate a target image from visually similar images. It introduces DifDisCap, a transformer-based framework that uses Group-based Differential Memory Attention (GDMA) and memory-difference encoding to highlight object features unique within an image group, guided by distinctive words and a Distinctive Word Rate metric. The method combines memory-based feature weighting with two specialized losses (WeiDisLoss and MemClsLoss) and a two-stage training regime to balance distinctiveness and accuracy. Empirical results on MSCOCO and Flickr30k show meaningful gains in distinctiveness with a modest trade-off in conventional accuracy, supported by ablations and human-in-the-loop evaluation. The approach enhances interpretability and could extend to related tasks such as distinctive video captioning and contrastive learning in groups of similar images.

Abstract

Recent advances in image captioning have focused on enhancing accuracy by substantially increasing the dataset and model size. While conventional captioning models exhibit high performance on established metrics such as BLEU, CIDEr, and SPICE, the capability of captions to distinguish the target image from other similar images is under-explored. To generate distinctive captions, a few pioneers employed contrastive learning or re-weighted the ground-truth captions. However, these approaches often overlook the relationships among objects in a similar image group (e.g., items or properties within the same album or fine-grained events). In this paper, we introduce a novel approach to enhance the distinctiveness of image captions, namely Group-based Differential Distinctive Captioning Method, which visually compares each image with other images in one similar group and highlights the uniqueness of each image. In particular, we introduce a Group-based Differential Memory Attention (GDMA) module, designed to identify and emphasize object features in an image that are uniquely distinguishable within its image group, i.e., those exhibiting low similarity with objects in other images. This mechanism ensures that such unique object features are prioritized during caption generation for the image, thereby enhancing the distinctiveness of the resulting captions. To further refine this process, we select distinctive words from the ground-truth captions to guide both the language decoder and the GDMA module. Additionally, we propose a new evaluation metric, the Distinctive Word Rate (DisWordRate), to quantitatively assess caption distinctiveness. Quantitative results indicate that the proposed method significantly improves the distinctiveness of several baseline models, and achieves state-of-the-art performance on distinctiveness while not excessively sacrificing accuracy...

Figures (6)

• Figure 1: Our model generates distinctive captions that can distinguish the target image from other similar images. Compared to current distinctive image captioning models such as DiscCap 2_disccap and CIDErBtwCap 17_wang2020compare, our captions can specify the important details, e.g., the color and the context of the traffic light, which can help a visually-impaired person to cross the street.
• Figure 2: Left: the standard transformer-based captioning model, where the target image features $X_0$ are the region-based visual features extracted via RoI pooling from Fast R-CNN. Right: our Group-based Differential Distinctive Captioning method (DifDisCap), which consists of a group-based differential memory attention (GDMA) module that weights the memory features according to their similarity with other similar images. The words in blue are distinctive words $w_d$, and the words with higher relatedness are marked in the darker color. Our model takes a group of images as input, and outputs one caption for each image. Only one target memory $M_0'$, one decoder, and one output caption are shown here to reduce clutter.
• Figure 3: The architecture of the Memory Difference Encoding module. The union memory vector $M_{u}$ encodes the fused information from images $I_0$ to $I_K$, and $\tilde{M}_0$ encodes the difference between the target image $I_0$ and other similar images.
• Figure 4: The trade-off between accuracy (CIDEr) and distinctiveness (DisWordRate): human-annotated GT captions (Human), baseline Transformer model (TF) 2_disccap, and three variants of our model using various components: image group based training (ImageGroup, IG), weighted distinctive loss (WeiDisLoss, WDL), group-based differential memory attention (GDMA), and Indicated Training (IndTrain, IT). For our models, we show three training stages (at different epochs), which demonstrates the trade-off between accuracy and distinctiveness during training.
• Figure 5: The user study interface. We display a group of six similar images, and a caption generated from one image by an image captioning model. The users are asked to select the image that they think the caption is describing.