Unified Multi-Dataset Training for TBPS

TL;DR

This work tackles the limitation of dataset-centric TBPS by proposing Scale-TBPS, a unified training framework that merges multiple TBPS datasets and scales to hundreds of thousands of identities. It introduces Noise-Aware Unified Dataset Curation (NDC) to robustly filter cross-dataset pairs using multiple pretrained experts, and a Discriminative Identity Learning (DIL) approach with a Multimodal Angular Identity loss to maintain discriminability across a vast identity space. The method combines MA-ID with a ranking loss and applies Test-time Nearest-Neighbor Normalization (NNN) to mitigate hubness, achieving superior or competitive results across CUHK-PEDES, ICFG-PEDES, RSTPReid, IIITD-20K, and UFine6926, often outperforming dataset-specific models. The findings demonstrate that a single, well-curated unified TBPS model can generalize across diverse distributions and even unseen domains, highlighting practical implications for scalable retrieval systems and cross-dataset deployment.

Abstract

Text-Based Person Search (TBPS) has seen significant progress with vision-language models (VLMs), yet it remains constrained by limited training data and the fact that VLMs are not inherently pre-trained for pedestrian-centric recognition. Existing TBPS methods therefore rely on dataset-centric fine-tuning to handle distribution shift, resulting in multiple independently trained models for different datasets. While synthetic data can increase the scale needed to fine-tune VLMs, it does not eliminate dataset-specific adaptation. This motivates a fundamental question: can we train a single unified TBPS model across multiple datasets? We show that naive joint training over all datasets remains sub-optimal because current training paradigms do not scale to a large number of unique person identities and are vulnerable to noisy image-text pairs. To address these challenges, we propose Scale-TBPS with two contributions: (i) a noise-aware unified dataset curation strategy that cohesively merges diverse TBPS datasets; and (ii) a scalable discriminative identity learning framework that remains effective under a large number of unique identities. Extensive experiments on CUHK-PEDES, ICFG-PEDES, RSTPReid, IIITD-20K, and UFine6926 demonstrate that a single Scale-TBPS model outperforms dataset-centric optimized models and naive joint training.

Figures (6)

• Figure 1: Illustration of Scale-TBPS. (a) illustrates the conventional dataset-centric training paradigm, where separate models are independently trained for different distributions, resulting in isolated models. (b) depicts naive joint training, where a single model is trained on merged datasets; however the learned representation fails to adequately accommodate samples from all distributions. (c) presents our proposed method, in which a single unified model is trained cohesively across distributions, effectively capturing shared semantics
• Figure 2: Overview of the proposed Scale-TBPS. (a) Noise-Aware Data Curation (NDC): Text--image pairs from the joint dataset ($\mathcal{D}$) are encoded using a set of pretrained and frozen models $\Phi$. top-$K$ retrieved samples are computed independently for each model. A pair is retained as a clean sample if it is ranked within the top-$K$ results by at least one pretrained model; such selected pairs are highlighted in green. (b) Discriminative Identity Learning (DIL): Filtered text--image pairs from $\mathcal{D}^{\text{clean}}$ are encoded by the learnable image and text encoders and encoded features $f^{v}$ and $f^{t}$ are passed through a shared multimodal class weight $w$. MA-ID denotes the proposed multimodal angular identity loss while $m$ denotes the margin
• Figure 3: t-SNE visualization comparing naive joint training and the proposed DIL. Different color signifies different IDs. Our method exhibits tighter intra-class clustering and improved inter-class separation
• Figure 4: Examples of noisy samples removed by our NDC module. Noisy parts are highlighted in red
• Figure 5: Data retention rates (%) after filtration by our NDC module across various values of $K$ for different TBPS datasets.