Image-Text Knowledge Modeling for Unsupervised Multi-Scenario Person Re-Identification

TL;DR

The paper tackles unsupervised person re-identification across multiple scenarios, introducing UMS-ReID and the ITKM framework that leverages vision-language models (CLIP) to fuse image and text representations. ITKM proceeds in three stages: Stage I uses a scenario-embedded, two-branch image encoder to learn homogeneous representations and pseudo-labels; Stage II learns text embeddings tied to pseudo-labels via image-text and text-image contrastive losses while enforcing inter-scenario separation with $L_{mss}$; Stage III aligns heterogeneous matches through CHM and IHM and maintains image-text consistency with the dynamic text representation update (DRU). Ablation studies validate each component—scenario embedding $e^s$, $L_{mss}$, DRU, CHM, and IHM—showing improved performance and strong generalization when training across three scenarios. Empirically, ITKM outperforms single-scenario methods and surpasses existing multi-scenario baselines, demonstrating the value of jointly leveraging diverse scenarios and cross-modal supervision for robust unsupervised ReID.

Abstract

We propose unsupervised multi-scenario (UMS) person re-identification (ReID) as a new task that expands ReID across diverse scenarios (cross-resolution, clothing change, etc.) within a single coherent framework. To tackle UMS-ReID, we introduce image-text knowledge modeling (ITKM) -- a three-stage framework that effectively exploits the representational power of vision-language models. We start with a pre-trained CLIP model with an image encoder and a text encoder. In Stage I, we introduce a scenario embedding in the image encoder and fine-tune the encoder to adaptively leverage knowledge from multiple scenarios. In Stage II, we optimize a set of learned text embeddings to associate with pseudo-labels from Stage I and introduce a multi-scenario separation loss to increase the divergence between inter-scenario text representations. In Stage III, we first introduce cluster-level and instance-level heterogeneous matching modules to obtain reliable heterogeneous positive pairs (e.g., a visible image and an infrared image of the same person) within each scenario. Next, we propose a dynamic text representation update strategy to maintain consistency between text and image supervision signals. Experimental results across multiple scenarios demonstrate the superiority and generalizability of ITKM; it not only outperforms existing scenario-specific methods but also enhances overall performance by integrating knowledge from multiple scenarios.

Figures (10)

• Figure 1: Identity information is scenario-dependent.
• Figure 2: The proposed ITKM framework for UMS-ReID consists of three stages. Stage I performs unsupervised homogeneous learning to generate pseudo-labels. Stage II learns text embeddings in the sentence "A photo of a ${[X_1]}{\rm{ }}[{X_2}]{\rm{ }}...{\rm{ }}[{X_M}]$ person" to associate it with the images for a homogeneous pseudo-label via CLIP. Stage III conducts unsupervised heterogeneous learning using CHM and IHM. In Stages I and III, colors indicate pseudo-labels and shapes indicate homogeneous groups.
• Figure 3: Architecture of image encoder used to obtain identity representations $f_{a,m}^{s,i}$ from an input image $x_{a,m}^s$.
• Figure 4: Two-dimensional t-SNE tsne visualizations of the text representations for ablated methods M2 and M3. Different colors representing text representations from different scenarios.
• Figure 5: F-score for M4 and M5.