CONQUER: Context-Aware Representation with Query Enhancement for Text-Based Person Search

TL;DR

CONQUER tackles Text-Based Person Search by addressing cross-modal gaps and query ambiguity through a two-stage design: Context-Aware Representation Enhancement (CARE) during training to enrich cross-modal embeddings with multi-granularity encoding, complementary pair mining, and context-guided Optimal Transport; and Interactive Query Enhancement (IQE) at inference to refine vague queries via anchor-based attribute enrichment and fusion for re-ranking. The CARE module learns robust embeddings by aligning local and global features with OT-guided matching and KL guidance, while IQE adaptively augments user queries using high-confidence candidate attributes without retraining the backbone. Empirical results on CUHK-PEDES, ICFG-PEDES, and RSTPReid show CONQUER achieving state-of-the-art performance and strong cross-domain robustness, with notable gains in incomplete-query scenarios. The work demonstrates practical TBPS improvements and provides a plug-and-play inference mechanism that can be deployed in real-world systems with publicly available code.

Abstract

Text-Based Person Search (TBPS) aims to retrieve pedestrian images from large galleries using natural language descriptions. This task, essential for public safety applications, is hindered by cross-modal discrepancies and ambiguous user queries. We introduce CONQUER, a two-stage framework designed to address these challenges by enhancing cross-modal alignment during training and adaptively refining queries at inference. During training, CONQUER employs multi-granularity encoding, complementary pair mining, and context-guided optimal matching based on Optimal Transport to learn robust embeddings. At inference, a plug-and-play query enhancement module refines vague or incomplete queries via anchor selection and attribute-driven enrichment, without requiring retraining of the backbone. Extensive experiments on CUHK-PEDES, ICFG-PEDES, and RSTPReid demonstrate that CONQUER consistently outperforms strong baselines in both Rank-1 accuracy and mAP, yielding notable improvements in cross-domain and incomplete-query scenarios. These results highlight CONQUER as a practical and effective solution for real-world TBPS deployment. Source code is available at https://github.com/zqxie77/CONQUER.

Figures (2)

• Figure 1: A comparison of TBPS.(a) Existing Method: The search is performed directly using the original text query.(b) Our IQE Approach: We improve the query at inference time without retraining. First, our method finds a relevant anchor image. Then, an MLLM learns key visual details from this image through a Q&A process. Finally, it fuses these details with the original text to create an improved query and re-ranks the search results.
• Figure 2: The Architecture of the Context-Aware Representation Enhancement (CARE) Module. The CARE module leverages multi-granularity representation encoding to learn robust cross-modal alignments. For complementary pair mining, it first classifies training pairs into clean, uncertain, and refinable sets by jointly analyzing similarity matrices from both global features and selected local tokens; hard negatives are then mined from the identified 'refinable' set. Simultaneously, its context-guided optimal matching component employs an Optimal Transport (OT) solver to align fine-grained features. This local alignment is in turn guided by the global similarities via a KL-Divergence loss, ensuring feature matching remains consistent with the overall semantic context.