CILP-FGDI: Exploiting Vision-Language Model for Generalizable Person Re-Identification

TL;DR

This work tackles generalizable person re-identification by harnessing vision-language models, specifically CLIP, to learn fine-grained and domain-invariant features. It introduces CLIP-FGDI, a three-stage training framework that first tunes the image encoder for fine-grained cues, then generates and leverages domain-invariant prompts via a bidirectional guiding mechanism, and finally refines with a composite loss including a novel apn triplet term. The approach demonstrates state-of-the-art performance across standard DG-ReID protocols and provides extensive ablations and analyses, including cross-model generalization to BLIP and backbone variants. The proposed method advances cross-domain robustness by integrating textual prompts as domain priors and enforcing domain-invariant guidance, with practical impact on surveillance and cross-camera person matching where domain shifts are prevalent.

Abstract

The Visual Language Model, known for its robust cross-modal capabilities, has been extensively applied in various computer vision tasks. In this paper, we explore the use of CLIP (Contrastive Language-Image Pretraining), a vision-language model pretrained on large-scale image-text pairs to align visual and textual features, for acquiring fine-grained and domain-invariant representations in generalizable person re-identification. The adaptation of CLIP to the task presents two primary challenges: learning more fine-grained features to enhance discriminative ability, and learning more domain-invariant features to improve the model's generalization capabilities. To mitigate the first challenge thereby enhance the ability to learn fine-grained features, a three-stage strategy is proposed to boost the accuracy of text descriptions. Initially, the image encoder is trained to effectively adapt to person re-identification tasks. In the second stage, the features extracted by the image encoder are used to generate textual descriptions (i.e., prompts) for each image. Finally, the text encoder with the learned prompts is employed to guide the training of the final image encoder. To enhance the model's generalization capabilities to unseen domains, a bidirectional guiding method is introduced to learn domain-invariant image features. Specifically, domain-invariant and domain-relevant prompts are generated, and both positive (pulling together image features and domain-invariant prompts) and negative (pushing apart image features and domain-relevant prompts) views are used to train the image encoder. Collectively, these strategies contribute to the development of an innovative CLIP-based framework for learning fine-grained generalized features in person re-identification.

Figures (6)

• Figure 1: Using the powerful cross-modal capabilities of the Vison-Language model, we employ a bidirectional guiding method, which involves fine-tuning the image encoder using features derived from prompts that are both domain-invariant and domain-relevant. This bidirectional guidance allows the model to learn domain-invariant features effectively.
• Figure 2: Overview of our method. In the first stage, a small number of epochs are employed to train the image encoder. In the second stage, the Gradient Reversal Layer (GRL) is utilized to learn domain-invariant ID-specific tokens and Domain-specific tokens. In the third stage, fine-tuning of the image encoder is performed using loss designed for downstream tasks. Here, "IF", "TF" and "TFd" represent "Image Feature", "Text Feature" and "Text Feature with domain information", respectively.
• Figure 3: Line charts illustrating the results for different loss functions and varying values of the hyper-parameter $\beta$.
• Figure 4: Line graph of experimental results for different initialization epochs.
• Figure 5: The t-SNE visualization of embeddings on the target domains: (a) t-SNE results obtained using baseline; (b) t-SNE results obtained using our method. (c), (d), (e) and (f) are scatter plots comparing the baselines with our method on PRID, GRID, VIPeR and iLIDs datasets, respectively. Best viewed in colors.