CLIP-DFGS: A Hard Sample Mining Method for CLIP in Generalizable Person Re-Identification

TL;DR

A hard sample mining method called Depth-First Graph Sampler (DFGS), based on depth-first search, designed to offer sufficiently challenging samples to enhance CLIP’s ability to extract fine-grained features and enhance the model’s ability to differentiate between individuals.

Abstract

Recent advancements in pre-trained vision-language models like CLIP have shown promise in person re-identification (ReID) applications. However, their performance in generalizable person re-identification tasks remains suboptimal. The large-scale and diverse image-text pairs used in CLIP's pre-training may lead to a lack or insufficiency of certain fine-grained features. In light of these challenges, we propose a hard sample mining method called DFGS (Depth-First Graph Sampler), based on depth-first search, designed to offer sufficiently challenging samples to enhance CLIP's ability to extract fine-grained features. DFGS can be applied to both the image encoder and the text encoder in CLIP. By leveraging the powerful cross-modal learning capabilities of CLIP, we aim to apply our DFGS method to extract challenging samples and form mini-batches with high discriminative difficulty, providing the image model with more efficient and challenging samples that are difficult to distinguish, thereby enhancing the model's ability to differentiate between individuals. Our results demonstrate significant improvements over other methods, confirming the effectiveness of DFGS in providing challenging samples that enhance CLIP's performance in generalizable person re-identification.

Figures (7)

• Figure 1: Three different sampling methods: (a) PK sampler; (b) Graph Sampler (GS); (c) Depth First Graph Sampler (DFGS). Each shape represents a different class, and each color represents a different batch.
• Figure 2: On one hand, the same individual captured by different cameras may exhibit significant differences due to variations in angles, backgrounds, resolutions, etc. Thus, we define individuals captured by different cameras as intra-class hard samples. On the other hand, in the training dataset, there may exist samples that are very similar but do not belong to the same individual, so we define these as inter-class hard samples.
• Figure 3: Overview of our method. Firstly, for each person ID, a specific text description is learned. Then, based on the acquired text descriptions, features are extracted and the pairwise distance similarity matrix is calculated and saved. Subsequently, during the sampling and learning stages, a sample graph is constructed using the pairwise distance similarity matrix. Through a depth-first search on this sample graph, training sample iterations are obtained, thus providing mini-batches containing challenging samples for fine-tuning the image encoder. Here, we use a directed graph for representing the structure, dashed lines indicate the directed edges of the graph, while solid lines represent the traversal sequence of the depth-first search.
• Figure 4: Depth-First Graph Sampler
• Figure 5: Parameter analysis: (a)(b)(c) represent the cases where $k$ is 5, 10 and 15.