DLCR: A Generative Data Expansion Framework via Diffusion for Clothes-Changing Person Re-ID

TL;DR

DLCR is proposed, a novel data expansion frame-work that employs diffusion-based text-guided inpainting, conditioned on clothing prompts constructed using LLMs, to generate synthetic data that only modifies a subject's clothes, while preserving their personally identifiable features.

Abstract

With the recent exhibited strength of generative diffusion models, an open research question is if images generated by these models can be used to learn better visual representations. While this generative data expansion may suffice for easier visual tasks, we explore its efficacy on a more difficult discriminative task: clothes-changing person re-identification (CC-ReID). CC-ReID aims to match people appearing in non-overlapping cameras, even when they change their clothes across cameras. Not only are current CC-ReID models constrained by the limited diversity of clothing in current CC-ReID datasets, but generating additional data that retains important personal features for accurate identification is a current challenge. To address this issue we propose DLCR, a novel data expansion framework that leverages pre-trained diffusion and large language models (LLMs) to accurately generate diverse images of individuals in varied attire. We generate additional data for five benchmark CC-ReID datasets (PRCC, CCVID, LaST, VC-Clothes, and LTCC) and increase their clothing diversity by 10X, totaling over 2.1M images generated. DLCR employs diffusion-based text-guided inpainting, conditioned on clothing prompts constructed using LLMs, to generate synthetic data that only modifies a subject's clothes while preserving their personally identifiable features. With this massive increase in data, we introduce two novel strategies - progressive learning and test-time prediction refinement - that respectively reduce training time and further boosts CC-ReID performance. On the PRCC dataset, we obtain a large top-1 accuracy improvement of 11.3% by training CAL, a previous state of the art (SOTA) method, with DLCR-generated data. We publicly release our code and generated data for each dataset here: https://github.com/CroitoruAlin/dlcr.

Figures (8)

• Figure 1: Best viewed with zoom. (Left) Comparing alternative data expansion approaches with our method. Simply changing the color of the clothes using our clothing masks preserves the subject's ID information, but lacks diversity. Using standard diffusion without inpainting can sometimes introduce diversity, but the ID-related information in an image is destroyed. Our method strongly increases clothing diversity, even with specific and difficult prompts, and preserves the subject's ID-related information for effective CC-ReID. (Right) DLCR significantly increases the size of training data for five benchmark CC-ReID datasets, generating over $2.1$M diverse, clothes-changed images.
• Figure 2: Overview of DLCR data generation (stage 1). Given a training image $x_i$, we apply an ID-preserving mask, $m_i$, where only the clothing regions are marked for inpainting. A target clothing ID, $c$, is randomly selected from the training dataset and a textual description of the clothing items belonging to $c$ is extracted. Specifically, we use LLaVA to extract frame-wise descriptions for every image in $X^c$, and use LLaMA to obtain a summarized description $R^c$. This description is used as a prompt for diffusion inpainting to artificially change a subject's clothing. Our ID-preserving diffusion inpainting model synthesizes quality generated data $\hat{x}_0$ for improved downstream CC-ReID training.
• Figure A3: Qualitative examples of our generated data for PRCC (row 1), LTCC (row 2) and CCVID (row 3) datasets. For each original image, we show three inpainted versions. The prompts used to generate the inpainted samples are placed under the corresponding images. These samples depict high-quality, diverse generated data that is prompt-aligned.
• Figure A4: Qualitative retrievals of CAL+DLCR versus baseline CAL. For a given query image, the top-3 retrieved images from the gallery are shown, with correct and incorrect retrievals outlined in green and red, respectively. Despite clothing changes between the query and gallery images, CAL+DLCR retrieves the correct subject regardless of appearance. However, CAL still favors clothing items during retrieval, often retrieving incorrect subjects from the gallery that share similar clothing items to the query. This shows that discriminative approaches to clothing-invariance, such as CAL, can still be further improved using generative methods.
• Figure A5: t-SNE visualizations of the query and gallery features produced by CAL+DLCR and CAL for $5$ randomly selected test subjects in the PRCC dataset. With a baseline CAL model (right), the query feature cluster for Subject $272$ are erroneously closer to the gallery feature cluster for Subject $4$, with the same issue between Subject $60$ and $56$. The incorrect clustering behaviors are marked with multi-colored boundaries. In contrast, the gallery and query feature clusters produced by CAL+DLCR (left) for the same subjects correctly cluster together, exhibiting DLCR's direct impact in learning better discriminative features and improving top-1 accuracy.