Multi-path Exploration and Feedback Adjustment for Text-to-Image Person Retrieval

TL;DR

This paper proposes MeFa, a Multi-Pathway Exploration, Feedback, and Adjustment framework, which deeply explores intrinsic feedback of intra and inter-modal to make targeted adjustment, thereby achieving more precise person-text associations.

Abstract

Text-based person retrieval aims to identify the specific persons using textual descriptions as queries. Existing ad vanced methods typically depend on vision-language pre trained (VLP) models to facilitate effective cross-modal alignment. However, the inherent constraints of VLP mod-els, which include the global alignment biases and insuffi-cient self-feedback regulation, impede optimal retrieval per formance. In this paper, we propose MeFa, a Multi-Pathway Exploration, Feedback, and Adjustment framework, which deeply explores intrinsic feedback of intra and inter-modal to make targeted adjustment, thereby achieving more precise person-text associations. Specifically, we first design an intra modal reasoning pathway that generates hard negative sam ples for cross-modal data, leveraging feedback from these samples to refine intra-modal reasoning, thereby enhancing sensitivity to subtle discrepancies. Subsequently, we intro duce a cross-modal refinement pathway that utilizes both global information and intermodal feedback to refine local in formation, thus enhancing its global semantic representation. Finally, the discriminative clue correction pathway incorpo rates fine-grained features of secondary similarity as discrim inative clues to further mitigate retrieval failures caused by disparities in these features. Experimental results on three public benchmarks demonstrate that MeFa achieves superior person retrieval performance without necessitating additional data or complex structures.

Figures (5)

• Figure 1: Illustration of MeFa Innovation: (a)Existing methods primarily rely on pre-trained models for feature extraction, using local, global, and cross-hard alignment techniques to explore correlations between textual descriptions and pedestrians. (b) MeFa enhances alignment accuracy and model robustness by constructing negative samples (IMR), refining features (CMR), and preserving discriminative clues (DCC).
• Figure 2: Overview of the proposed MeFa.It utilize the EvaCLIP encoder to extract textual and image features. The model's capability for fine-grained association is enhanced through three pathways: 1) Intra-modal inference pathway refines sensitivity to subtle variations using minimally different negative samples; 2) Cross-modal refinement pathway progressively transitions feature refinement from global to local; and 3) Discriminative clue correction pathway rectifies person-text mismatches caused by subtle discrepancies.
• Figure 3: It displays Rank-10 qualitative retrieval results for MeFa versus baseline models, arranged in descending similarity from right to left. Correct matches are marked in red, with green and pink annotations highlighting precise details and actions successfully identified by MeFa in the image descriptions.
• Figure 4: The similarity between part-of-speech segmented words and person images.
• Figure 5: Ablation analysis of MeFa on fine-grained information. The baseline (Mask) comparison group demonstrates the effect of masking the top three high-frequency nouns in the test data, independently assessing the impact on fine-grained textual descriptions.