Diverse Semantics-Guided Feature Alignment and Decoupling for Visible-Infrared Person Re-Identification

TL;DR

Propot introduces a prototypical prompting framework for text-to-image person re-identification, leveraging CLIP to initialize identity prototypes and a sequence of prompting modules to create diverse, task-adapted representations. Domain-conditional prompting (DPP) and instance-conditional prompting (IPP) generate multiple identity-enriched prototypes, which are adaptively aggregated (APA) and diffused to instances via prototype-to-instance contrastive learning, alongside an auxiliary masked language modeling objective. The framework jointly optimizes instance-level cross-modal alignment and identity-level matching, achieving strong results across CUHK-PEDES, ICFG-PEDES, and RSTPReid, notably state-of-the-art on ICFG-PEDES and competitive on the others. Propot’s end-to-end design and its use of identity-enriched prototypes demonstrate the effectiveness of combining vision-language priors with prompt-based adaptation for robust TIReID performance.

Abstract

Visible-Infrared Person Re-Identification (VI-ReID) is a challenging task due to the large modality discrepancy between visible and infrared images, which complicates the alignment of their features into a suitable common space. Moreover, style noise, such as illumination and color contrast, reduces the identity discriminability and modality invariance of features. To address these challenges, we propose a novel Diverse Semantics-guided Feature Alignment and Decoupling (DSFAD) network to align identity-relevant features from different modalities into a textual embedding space and disentangle identity-irrelevant features within each modality. Specifically, we develop a Diverse Semantics-guided Feature Alignment (DSFA) module, which generates pedestrian descriptions with diverse sentence structures to guide the cross-modality alignment of visual features. Furthermore, to filter out style information, we propose a Semantic Margin-guided Feature Decoupling (SMFD) module, which decomposes visual features into pedestrian-related and style-related components, and then constrains the similarity between the former and the textual embeddings to be at least a margin higher than that between the latter and the textual embeddings. Additionally, to prevent the loss of pedestrian semantics during feature decoupling, we design a Semantic Consistency-guided Feature Restitution (SCFR) module, which further excavates useful information for identification from the style-related features and restores it back into the pedestrian-related features, and then constrains the similarity between the features after restitution and the textual embeddings to be consistent with that between the features before decoupling and the textual embeddings. Extensive experiments on three VI-ReID datasets demonstrate the superiority of our DSFAD.

Figures (4)

• Figure 1: The motivation of our proposed Propot. (a) Some examples of TIReID data containing multiple images from two identities and their annotated texts. Instances under the same identity are significantly different. (b) Most existing TIReID methods only focus on instance-level matching and ignore identity-level matching. (c) Our Propot proposes a prototype prompting framework to produce identity-enriched prototypes and diffuse their rich identity information to instances for modeling identity-level matching.
• Figure 2: Overview of our Propot. It includes instance-level matching and identity-enriched prototype learning. For instance-level matching, each image and its annotated text are directly aligned through SDM loss (Baseline). For prototype learning, we first utilize pre-trained CLIP to generate the initial prototypes ($\bm {pt}^v$ and $\bm {pt}^t$). We then adapt the initial prototypes to TIReID through the DPP module to generate the task-adapted prototypes ($\bm {p}_a^v$ and $\bm {p}_a^t$). And the IPP module updates the prototypes conditioned on a batch of intra-modal and inter-modal instances to generate intra-modal and inter-modal enriched prototypes ($\bm {p}_{en}^v$, $\bm {p}_{en}^t$, $\bm {p}_{eo}^v$ and $\bm {p}_{eo}^t$), respectively. The above multiple prototypes are aggregated through Adaptive Prototypical Aggregation (APA) to generate the final prototypes ($\bm {p}^v$ and $\bm {p}^t$), and their rich identity information is diffused to each instance through prototype-to-instance contrastive loss ($\mathcal{L}_{p2v}$, $\mathcal{L}_{p2t}$) to model identity-level matching. Moreover, we also introduce the MLM module as compensation to model fine-grained matching. During testing, only visual and textual encoders are used for inference.
• Figure 3: Effects of four hyper-parameters on CUHK-PEDES, including contextual vector length $K$, the block number $N_a, N_e$, and loss weight $\lambda_1$.
• Figure 4: Retrieval result comparisons of Baseline and our Propot on CUHK-PEDES. The matched and mismatched person images are marked with green and red rectangles, respectively.