CLIP-SCGI: Synthesized Caption-Guided Inversion for Person Re-Identification

TL;DR

This work tackles the challenge of relying on implicit text embeddings for person re-identification by generating explicit captions for training images using LLAVA and integrating them into a CLIP-based framework. Through a caption-guided inversion (CGI) and a contextual feature fusion (CFF) module, CLIP-SCGI leverages synthesized captions to guide the learning of discriminative, robust representations without adding inference cost. Extensive experiments across Market-1501, MSMT17, DukeMTMC-reID, and Occluded-Duke show state-of-the-art gains, with notable improvements in mAP and Rank-1, and ablations validate the importance of both CGI and CFF. The approach provides a practical, one-stage training pipeline that capitalizes on cross-modal supervision while keeping inference efficient, though it relies on offline caption generation, a gap future work aims to close by moving toward implicit caption-free methods.

Abstract

Person re-identification (ReID) has recently benefited from large pretrained vision-language models such as Contrastive Language-Image Pre-Training (CLIP). However, the absence of concrete descriptions necessitates the use of implicit text embeddings, which demand complicated and inefficient training strategies. To address this issue, we first propose one straightforward solution by leveraging existing image captioning models to generate pseudo captions for person images, and thereby boost person re-identification with large vision language models. Using models like the Large Language and Vision Assistant (LLAVA), we generate high-quality captions based on fixed templates that capture key semantic attributes such as gender, clothing, and age. By augmenting ReID training sets from uni-modality (image) to bi-modality (image and text), we introduce CLIP-SCGI, a simple yet effective framework that leverages synthesized captions to guide the learning of discriminative and robust representations. Built on CLIP, CLIP-SCGI fuses image and text embeddings through two modules to enhance the training process. To address quality issues in generated captions, we introduce a caption-guided inversion module that captures semantic attributes from images by converting relevant visual information into pseudo-word tokens based on the descriptions. This approach helps the model better capture key information and focus on relevant regions. The extracted features are then utilized in a cross-modal fusion module, guiding the model to focus on regions semantically consistent with the caption, thereby facilitating the optimization of the visual encoder to extract discriminative and robust representations. Extensive experiments on four popular ReID benchmarks demonstrate that CLIP-SCGI outperforms the state-of-the-art by a significant margin.

Figures (7)

• Figure 1: Comparison between (a) the previous CLIP-ReID method, where the text prompt in the grey-highlighted area indicates learnable embeddings requiring pretraining first, and (b) our proposed CLIP-SCGI method, which combines explicitly generated text embeddings with image features for guidance.
• Figure 2: Caption Generation results. The first row shows the captions generated for the Market1501 dataset, while the second row depicts the results for the MSMT17 dataset.
• Figure 3: Overview of CLIP-SCGI. It enhances CLIP with two core modules named caption-guided inversion(CGI) and contextual feature fusion(CFF) for ReID. The CGI module transforms the images to a pseudo-word under the guidance of captions, while the CFF module fuses the pseudo-word embeddings and images to facilitate learning a discriminative global representation.
• Figure 4: The impact of the hyper-parameters at R1 and mAP on Market-1501, including (a) effects of CGI depth(left) ; (b) the number of learnable queries(right). The dark blue line represents R1, while the light blue line represents mAP.
• Figure 5: Comparison of top-10 retrieved results on MSMT17 dataset between our method (the first row) and CLIP-ReID (the second row) for each query. The incorrectly identified samples are highlighted in red.