GEA: Generation-Enhanced Alignment for Text-to-Image Person Retrieval

TL;DR

The paper tackles Text-to-Image Person Retrieval (TIPR) challenges posed by incomplete textual descriptions and limited data diversity. It introduces Generation-Enhanced Alignment (GEA), a framework with two modules: Text-Guided Token Enhancement (TGTE), which generates diffusion-based intermediate images conditioned on text to enrich semantics, and Generative Intermediate Fusion (GIF), which fuses information from the original image, the text, and the generated image via dual cross-attention and a transformer, trained with a triplet alignment loss. The backbone uses CLIP, enabling strong cross-modal priors, while training leverages standard datasets (CUHK-PEDES, RSTPReid, ICFG-PEDES) and evaluation metrics like Rank-1/5/10 and mAP. Across three benchmarks, GEA delivers competitive or superior performance, with notable gains in ranking and mAP, demonstrating improved cross-modal alignment and generalization in low-data scenarios.

Abstract

Text-to-Image Person Retrieval (TIPR) aims to retrieve person images based on natural language descriptions. Although many TIPR methods have achieved promising results, sometimes textual queries cannot accurately and comprehensively reflect the content of the image, leading to poor cross-modal alignment and overfitting to limited datasets. Moreover, the inherent modality gap between text and image further amplifies these issues, making accurate cross-modal retrieval even more challenging. To address these limitations, we propose the Generation-Enhanced Alignment (GEA) from a generative perspective. GEA contains two parallel modules: (1) Text-Guided Token Enhancement (TGTE), which introduces diffusion-generated images as intermediate semantic representations to bridge the gap between text and visual patterns. These generated images enrich the semantic representation of text and facilitate cross-modal alignment. (2) Generative Intermediate Fusion (GIF), which combines cross-attention between generated images, original images, and text features to generate a unified representation optimized by triplet alignment loss. We conduct extensive experiments on three public TIPR datasets, CUHK-PEDES, RSTPReid, and ICFG-PEDES, to evaluate the performance of GEA. The results justify the effectiveness of our method. More implementation details and extended results are available at https://github.com/sugelamyd123/Sup-for-GEA.

Figures (5)

• Figure 1: Comparison between incomplete and complete textual descriptions. (a) shows a person image with multiple details highlighted in red boxes (such as coat, jeans, shoes), which were not reflected by incomplete text query, while (b) presents a more complete description. This clearly illustrates how the lack of complete textual exacerbates the modality gap between visual and textual modalities in real-world TIPR scenarios.
• Figure 2: Framework of the GEA method. TGTE is responsible for generating and encoding intermediate images and extracts features from the input text, original image, and a diffusion-generated intermediate image, while GIF facilitates cross-modal interaction and implicit relational inference across the three representations via cross fusion. (a) is the illustration of the TGTE, which shows three parallel branches that extract features from the original image, the input text, and a diffusion-generated image conditioned on the text prompt. The generated image acts as an intermediate modality to enrich textual semantics. (b) illustrates the Generative Intermediate Fusion (GIF) module, where cross-attention mechanisms align the generated features with the original image and text features, respectively. (c) displays the core idea of our approach with three identities, marked in red, blue, and yellow. Our method pulls matched image-text features closer and separates non-matching ones.
• Figure 3: t-SNE visualization of feature embeddings. (a) Visualization of 10 samples with complete textual descriptions and 10 with incomplete descriptions. The fused tokens (circle) consistently lie between their corresponding text (square) and image tokens (star), indicating our method’s ability to bridge the modality gap regardless of text completeness. (b) Visualization of all 67 samples from five randomly selected identities. Our method effectively clusters corresponding image-text pairs while separating unrelated samples.
• Figure 4: Attention heatmap visualization comparing the baseline CLIP model and our proposed method. For each input pair, the top row shows the original image and its corresponding text description. The middle row displays the attention map from the baseline model, while the bottom row shows the attention map produced by our method.
• Figure 5: Variation of performance with different fusion weight.

Theorems & Definitions (2)

• Theorem 1: Fermat, 1637
• proof