Cross-modal Full-mode Fine-grained Alignment for Text-to-Image Person Retrieval

TL;DR

FMFA tackles TIPR by enforcing full-mode cross-modal alignment through explicit fine-grained interaction and adaptive implicit reasoning. It introduces A-SDM to focus on unmatched positive pairs and EFA to explicitly aggregate token and patch features via a sparse similarity matrix with hard coding. The approach achieves state-of-the-art performance among global matching methods on CUHK-PEDES, ICFG-PEDES, and RSTPReid while maintaining efficient inference. These results demonstrate robust cross-modal alignment across diverse encoders and pre-training regimes, offering practical benefits for text-to-image person retrieval.

Abstract

Text-to-Image Person Retrieval (TIPR) is a cross-modal matching task designed to identify the person images that best correspond to a given textual description. The key difficulty in TIPR is to realize robust correspondence between the textual and visual modalities within a unified latent representation space. To address this challenge, prior approaches incorporate attention mechanisms for implicit cross-modal local alignment. However, they lack the ability to verify whether all local features are correctly aligned. Moreover, existing methods tend to emphasize the utilization of hard negative samples during model optimization to strengthen discrimination between positive and negative pairs, often neglecting incorrectly matched positive pairs. To mitigate these problems, we propose FMFA, a cross-modal Full-Mode Fine-grained Alignment framework, which enhances global matching through explicit fine-grained alignment and existing implicit relational reasoning -- hence the term ``full-mode'' -- without introducing extra supervisory signals. In particular, we propose an Adaptive Similarity Distribution Matching (A-SDM) module to rectify unmatched positive sample pairs. A-SDM adaptively pulls the unmatched positive pairs closer in the joint embedding space, thereby achieving more precise global alignment. Additionally, we introduce an Explicit Fine-grained Alignment (EFA) module, which makes up for the lack of verification capability of implicit relational reasoning. EFA strengthens explicit cross-modal fine-grained interactions by sparsifying the similarity matrix and employs a hard coding method for local alignment. We evaluate our method on three public datasets, where it attains state-of-the-art results among all global matching methods. The code for our method is publicly accessible at https://github.com/yinhao1102/FMFA.

Figures (8)

• Figure 1: Evolution of global matching methods for text-to-image person retrieval. (a) Existing global matching methods focus on hard negative samples to learn a discriminative boundary in a common latent space, thus enhancing the distinction between positive and negative samples. (b) Our adaptive global matching method builds on a discriminative boundary and concentrates on unmatched positive samples, adaptively pulling them closer to the corresponding query text.
• Figure 2: The illustration of our FMFA framework. (a) Our FMFA contains a two-stream feature extraction network and four distinct modules for structured representation learning, namely Identity Identification (ID loss), Adaptive Similarity Distribution Matching (A-SDM), Explicit Fine-grained Alignment (EFA) and Implicit Relation Reasoning (IRR). The former two are coarse-grained alignment modules and the latter two are fine-grained alignment modules. Modules linked via dashed connections are omitted during inference. (b) A-SDM obtains the adaptive weight through the similarity score of unmatched positive pairs, adaptively pulling positive pairs closer and dynamically adjusting the pulling force. (c) EFA achieves fine-grained interaction by hard coding alignment of token or patch embeddings and their corresponding language-grouped vision embedding, which is derived from an aggregated sparse similarity matrix.
• Figure 3: Illustration of the EFA module. (a) EFA imposes a sparse structure on the similarity matrix relating token embeddings to patch embeddings, and obtains the language-grouped vision embeddings by aggregating the sparse similarity with its corresponding patch. (b) EFA obtains hard similarity through hard coding and LSE pooling, where the calculated hard similarity reflects the relationship between the language-grouped vision embeddings and their original token or patch embeddings.
• Figure 4: The sensitivity analysis of hyperparameters of FMFA on the CUHK-PEDES dataset.
• Figure 5: Visualization of similarity maps before and after the sparse process in the EFA module.