Attribute-Aware Implicit Modality Alignment for Text Attribute Person Search

TL;DR

The paper tackles text attribute person search by bridging the modality gap between text and images. It introduces AIMA, a CLIP-based framework with a Masked Attribute Prediction task for implicit local alignment and an Attribute-IoU Guided Intra-Modal Contrastive loss to phase attribute distributions in embedding space. Empirical results on Market-1501 Attribute, PETA, and PA100K show substantial improvements over state-of-the-art methods, validating the effectiveness of prompting, MAP, and IoU-guided supervision. The approach enables robust cross-modal matching and fine-grained semantic organization, offering strong practical impact for surveillance and descriptive search tasks.

Abstract

Text attribute person search aims to find specific pedestrians through given textual attributes, which is very meaningful in the scene of searching for designated pedestrians through witness descriptions. The key challenge is the significant modality gap between textual attributes and images. Previous methods focused on achieving explicit representation and alignment through unimodal pre-trained models. Nevertheless, the absence of inter-modality correspondence in these models may lead to distortions in the local information of intra-modality. Moreover, these methods only considered the alignment of inter-modality and ignored the differences between different attribute categories. To mitigate the above problems, we propose an Attribute-Aware Implicit Modality Alignment (AIMA) framework to learn the correspondence of local representations between textual attributes and images and combine global representation matching to narrow the modality gap. Firstly, we introduce the CLIP model as the backbone and design prompt templates to transform attribute combinations into structured sentences. This facilitates the model's ability to better understand and match image details. Next, we design a Masked Attribute Prediction (MAP) module that predicts the masked attributes after the interaction of image and masked textual attribute features through multi-modal interaction, thereby achieving implicit local relationship alignment. Finally, we propose an Attribute-IoU Guided Intra-Modal Contrastive (A-IoU IMC) loss, aligning the distribution of different textual attributes in the embedding space with their IoU distribution, achieving better semantic arrangement. Extensive experiments on the Market-1501 Attribute, PETA, and PA100K datasets show that the performance of our proposed method significantly surpasses the current state-of-the-art methods.

Figures (6)

• Figure 1: Illustration of the text attribute person search task, which aims to identify the images which match the attribute query.
• Figure 2: The overview of our proposed AIMA framework. It consists of an image encoder, a text encoder, and a multimodal encoder. It has four representation learning branches: Masked Attribute Prediction (MAP), Attribute-IoU Guided Intra-Modal Contrastive loss (A-IoU IMC), Similarity Distribution Matching (SDM), and Identity loss (ID loss). AIMA achieves end-to-end training through these four tasks, requiring only the computation of global feature representations during inference.
• Figure 3: Illustration of the calculation process of Attribute-IoU.
• Figure 4: Visualization results of attention weights on Market-1501 Attribute.
• Figure 5: t-SNE visualization of a part of the joint embedding space on Market-1501 Attribute. Stars and triangles indicate embedding vectors of texts and their associated images, respectively, and their colors mean their categories.