Modeling Thousands of Human Annotators for Generalizable Text-to-Image Person Re-identification

TL;DR

This work tackles the limited diversity of MLLM-generated captions for text-to-image ReID by introducing Human Annotator Modeling (HAM), which learns to mimic thousands of human description styles through clustering of CLIP-based style features and learnable style prompts. It further augments diversity with Uniform Prototype Sampling (UPS), which expands the style prototype space to cover a broader range of annotator preferences. The authors build HAM-PEDES, a large-scale annotated dataset, and demonstrate that HAM-PEDES improves generalization of ReID models in direct transfer and fine-tuning scenarios, achieving state-of-the-art results on several benchmarks. The approach reduces labeling costs and enhances cross-domain robustness, though it notes potential noise and hallucination from MLLMs as future work to address.

Abstract

Text-to-image person re-identification (ReID) aims to retrieve the images of an interested person based on textual descriptions. One main challenge for this task is the high cost in manually annotating large-scale databases, which affects the generalization ability of ReID models. Recent works handle this problem by leveraging Multi-modal Large Language Models (MLLMs) to describe pedestrian images automatically. However, the captions produced by MLLMs lack diversity in description styles. To address this issue, we propose a Human Annotator Modeling (HAM) approach to enable MLLMs to mimic the description styles of thousands of human annotators. Specifically, we first extract style features from human textual descriptions and perform clustering on them. This allows us to group textual descriptions with similar styles into the same cluster. Then, we employ a prompt to represent each of these clusters and apply prompt learning to mimic the description styles of different human annotators. Furthermore, we define a style feature space and perform uniform sampling in this space to obtain more diverse clustering prototypes, which further enriches the diversity of the MLLM-generated captions. Finally, we adopt HAM to automatically annotate a massive-scale database for text-to-image ReID. Extensive experiments on this database demonstrate that it significantly improves the generalization ability of ReID models.

Figures (3)

• Figure 1: Illustration of different schemes for enhancing the style diversity for MLLM-generated textual descriptions. (Top) Existing works typically rely on manually designed description templates to instruct MLLM to generate style-diverse captions. (Bottom) Our approach models the description styles of thousands of human annotators, and enables the MLLM to capture these styles in a learning-based approach.
• Figure 2: Overview of our framework. In HAM, we first extract style features from human annotations and then perform clustering on these features, as illustrated in (a). This allows us to group human annotations into $K_1+K_2$ clusters, each of which contain descriptions with similar styles. We employ $K_1+K_2$ prompts to represent these description styles. (b) illustrates that we employ prompt learning to model the description preference of different annotators with MLLM. To fully explore the diverse styles in human annotations, we propose UPS, a clustering method that complements KMeans as shown in (c).
• Figure 3: Pre-training data size’s impact on direct transfer ReID performance. Best viewed with zoom-in.