Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AMNS: Attention-Weighted Selective Mask and Noise Label Suppression for Text-to-Image Person Retrieval

Runqing Zhang, Xue Zhou

TL;DR

This work tackles noisy image-text correspondences and masking-induced semantic loss in text-to-image person retrieval. It presents AMNS, integrating an EMA-based Attention-Weighted Selective Mask with noise-suppressing Bidirectional Similarity Distribution Matching (BSDM) and a hard-sample focused Weight Adjustment Focal (WAF) loss to robustly align cross-modal representations. Key contributions include the introduction of BSDM for bidirectional similarity enforcement, the AWM strategy to preserve semantically relevant image regions during masking, and the integration of WAF, all validated across CUHK-PEDES, ICFG-PEDES, and RSTPReid with improvements in Rank-K, mAP, and mINP metrics. The approach enhances robustness to noisy annotations and masking distortions, enabling more reliable text-to-image pedestrian retrieval in real-world, noisy data contexts.

Abstract

Most existing text-to-image person retrieval methods usually assume that the training image-text pairs are perfectly aligned; however, the noisy correspondence(NC) issue (i.e., incorrect or unreliable alignment) exists due to poor image quality and labeling errors. Additionally, random masking augmentation may inadvertently discard critical semantic content, introducing noisy matches between images and text descriptions. To address the above two challenges, we propose a noise label suppression method to mitigate NC and an Attention-Weighted Selective Mask (AWM) strategy to resolve the issues caused by random masking. Specifically, the Bidirectional Similarity Distribution Matching (BSDM) loss enables the model to effectively learn from positive pairs while preventing it from over-relying on them, thereby mitigating the risk of overfitting to noisy labels. In conjunction with this, Weight Adjustment Focal (WAF) loss improves the model's ability to handle hard samples. Furthermore, AWM processes raw images through an EMA version of the image encoder, selectively retaining tokens with strong semantic connections to the text, enabling better feature extraction. Extensive experiments demonstrate the effectiveness of our approach in addressing noise-related issues and improving retrieval performance.

AMNS: Attention-Weighted Selective Mask and Noise Label Suppression for Text-to-Image Person Retrieval

TL;DR

This work tackles noisy image-text correspondences and masking-induced semantic loss in text-to-image person retrieval. It presents AMNS, integrating an EMA-based Attention-Weighted Selective Mask with noise-suppressing Bidirectional Similarity Distribution Matching (BSDM) and a hard-sample focused Weight Adjustment Focal (WAF) loss to robustly align cross-modal representations. Key contributions include the introduction of BSDM for bidirectional similarity enforcement, the AWM strategy to preserve semantically relevant image regions during masking, and the integration of WAF, all validated across CUHK-PEDES, ICFG-PEDES, and RSTPReid with improvements in Rank-K, mAP, and mINP metrics. The approach enhances robustness to noisy annotations and masking distortions, enabling more reliable text-to-image pedestrian retrieval in real-world, noisy data contexts.

Abstract

Most existing text-to-image person retrieval methods usually assume that the training image-text pairs are perfectly aligned; however, the noisy correspondence(NC) issue (i.e., incorrect or unreliable alignment) exists due to poor image quality and labeling errors. Additionally, random masking augmentation may inadvertently discard critical semantic content, introducing noisy matches between images and text descriptions. To address the above two challenges, we propose a noise label suppression method to mitigate NC and an Attention-Weighted Selective Mask (AWM) strategy to resolve the issues caused by random masking. Specifically, the Bidirectional Similarity Distribution Matching (BSDM) loss enables the model to effectively learn from positive pairs while preventing it from over-relying on them, thereby mitigating the risk of overfitting to noisy labels. In conjunction with this, Weight Adjustment Focal (WAF) loss improves the model's ability to handle hard samples. Furthermore, AWM processes raw images through an EMA version of the image encoder, selectively retaining tokens with strong semantic connections to the text, enabling better feature extraction. Extensive experiments demonstrate the effectiveness of our approach in addressing noise-related issues and improving retrieval performance.
Paper Structure (9 sections, 8 equations, 3 figures, 3 tables)

This paper contains 9 sections, 8 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. METHOD
  3. Attention-Weighted Selective Mask
  4. Noise Label Suppression Methods
  5. Experimental section
  6. Datasets and Settings
  7. Experimental results
  8. Ablation experiments
  9. Conclusion

Figures (3)

  • Figure 1: Overview of our proposed framework AMNS.
  • Figure 2: BSDM diagram. It demonstrates the core idea of implementing noisy label suppression. This approach constrains the true label distribution through the predicted distribution, preventing the model from becoming overconfident and mitigating the impact of noisy labels.
  • Figure 3: Rank-1 performance changes of the two strategies at different mask ratios.