ITSELF: Attention Guided Fine-Grained Alignment for Vision-Language Retrieval

TL;DR

ITSELF addresses fine-grained vision-language retrieval for text-based person search by turning encoder attention into an Attentive Bank for implicit local alignment. The core innovations are GRAB, which uses Multi-layer Attention for Robust Selection (MARS) to form a diverse, high-saliency token bank, and Adaptive Token Scheduler (ATS) to progressively focus on discriminative details during training. The method couples a global alignment loss with a local alignment objective, yielding a final similarity that blends coarse and fine-grained cues. Experiments on CUHK-PEDES, ICFG-PEDES, and RSTPReid show state-of-the-art performance among CLIP-based approaches and strong cross-dataset generalization, highlighting the practicality of attention-guided locality without external supervision.

Abstract

Vision Language Models (VLMs) have rapidly advanced and show strong promise for text-based person search (TBPS), a task that requires capturing fine-grained relationships between images and text to distinguish individuals. Previous methods address these challenges through local alignment, yet they are often prone to shortcut learning and spurious correlations, yielding misalignment. Moreover, injecting prior knowledge can distort intra-modality structure. Motivated by our finding that encoder attention surfaces spatially precise evidence from the earliest training epochs, and to alleviate these issues, we introduceITSELF, an attention-guided framework for implicit local alignment. At its core, Guided Representation with Attentive Bank (GRAB) converts the model's own attention into an Attentive Bank of high-saliency tokens and applies local objectives on this bank, learning fine-grained correspondences without extra supervision. To make the selection reliable and non-redundant, we introduce Multi-Layer Attention for Robust Selection (MARS), which aggregates attention across layers and performs diversity-aware top-k selection; and Adaptive Token Scheduler (ATS), which schedules the retention budget from coarse to fine over training, preserving context early while progressively focusing on discriminative details. Extensive experiments on three widely used TBPS benchmarks showstate-of-the-art performance and strong cross-dataset generalization, confirming the effectiveness and robustness of our approach without additional prior supervision. Our project is publicly available at https://trhuuloc.github.io/itself

Figures (12)

• Figure 1: Evolution of text-based person search paradigms. (a) Global matching method uses powerful MLLM to synthesize extra datasets (b) Recent local implicit matching method implicitly reasons over relations among all local tokens. (c) Ours - ITSELF with GRAB: an attention-guided local branch to learn implicitly fine-grained, discriminative features to achieve better alignment.
• Figure 2: Rank-1 accuracy gap between unmasked and masked images under different mask retention ratios during the early training epochs on RSTP dataset.
• Figure 3: Overview of our proposed ITSELF (an attention-guided implicit local alignment framework). The architecture features a dual-stream encoder for images (left) and text (right). At its core is the GRAB (Guided Representation with Attentive Bank) module, which is designed to learn fine-grained, discriminative cues. GRAB is composed of two key components: MARS (Multi-layer Attention for Robust Selection), which fuses attention across layers to select informative patches/tokens, and ATS (Adaptive Token Scheduler), which anneals the token selection from coarse to fine during training. The model is optimized with a dual-loss strategy: a local loss $L_{local}$ aligns the guided local representations, and a global loss $L_{global}$ matches the final overall embeddings. This allows ITSELF to reinforce global text-image alignment without requiring additional supervision or adding any inference-time cost.
• Figure 4: Qualitative results of text-to-image retrieval on RSTPReid benchmark, comparing our method with RDE rde. Retrieved images are ranked from left to right in descending order of similarity. Correct matches are outlined in green, while incorrect ones are shown in red. Text highlighted in green indicates the descriptive details effectively captured by our approach.
• Figure 5: Qualitative comparison of attention maps generated by RDE rde and by our method using the Grad-CAM.gradcam