Beat: Bi-directional One-to-Many Embedding Alignment for Text-based Person Retrieval

TL;DR

Beat tackles cross-modal gaps in text-based person retrieval by introducing a bi-directional embedding paradigm that projects each modality into its own space and generates multiple embeddings per sample via REM-G, enabling robust one-to-many alignment without adding trainable parameters. The approach combines a multi-grained feature extractor with dual losses (ID and CR) and sums similarities across M embedded variants in two modality-specific spaces, yielding state-of-the-art results on CUHK-PEDES, ICFG-PEDES, and RSTPReID while generalizing to MS-COCO, CUB, and Flowers. Key innovations include fixing one modality while aligning the other, and a bottleneck-based REM-G that produces multiple embeddings through M residual modules, addressing both optimization direction issues and one-to-many relations. Overall, Beat advances cross-modal retrieval by enabling more accurate, flexible alignment between text and person images, with practical implications for scalable, language-driven person search and related multi-modal tasks.

Abstract

Text-based person retrieval (TPR) is a challenging task that involves retrieving a specific individual based on a textual description. Despite considerable efforts to bridge the gap between vision and language, the significant differences between these modalities continue to pose a challenge. Previous methods have attempted to align text and image samples in a modal-shared space, but they face uncertainties in optimization directions due to the movable features of both modalities and the failure to account for one-to-many relationships of image-text pairs in TPR datasets. To address this issue, we propose an effective bi-directional one-to-many embedding paradigm that offers a clear optimization direction for each sample, thus mitigating the optimization problem. Additionally, this embedding scheme generates multiple features for each sample without introducing trainable parameters, making it easier to align with several positive samples. Based on this paradigm, we propose a novel Bi-directional one-to-many Embedding Alignment (Beat) model to address the TPR task. Our experimental results demonstrate that the proposed Beat model achieves state-of-the-art performance on three popular TPR datasets, including CUHK-PEDES (65.61 R@1), ICFG-PEDES (58.25 R@1), and RSTPReID (48.10 R@1). Furthermore, additional experiments on MS-COCO, CUB, and Flowers datasets further demonstrate the potential of Beat to be applied to other image-text retrieval tasks.

Figures (6)

• Figure 1: (a) Two possible approaches for the bi-directional search problem. The left-hand side illustrates both agents moving toward each other to meet at a midpoint, while the right-hand side shows one agent remaining stationary while the other is movable. (b) A one-to-many relationship of image-text pairs in the TPR dataset.
• Figure 2: Illustration of the proposed Beat model. Images and texts are first processed by ConvNet and SeqModel to obtain visual features $\bm{V}$ and textual features $\bm{T}$, respectively. Global features $\boldsymbol{v}_g$ and $\boldsymbol{t}_g$ are extracted through a global max pooling (GMP) and a fully connected layer (FC) based on $\bm{V}$ and $\bm{T}$. Similarly, we adopt a global max pooling (GMP) and an FC layer to obtain $K$ local visual features $\{\boldsymbol{v}_l^k\}|_{k=1}^K$. Then, we employ word attention module (WAM) ding2021semantically, GMP and FC layer to obtain $K$ local textual features $\{\boldsymbol{t}_l^k\}|_{k=1}^K$. We then adopt a non-local module (NLM) to get $K$ non-local visual features $\{\boldsymbol{v}_n^k\}|_{k=1}^K$ and textual features $\{\boldsymbol{t}_n^k\}|_{k=1}^K$. Afterward, the REM-G is introduced to perform bi-directional one-to-many embedding. Finally, textual and visual samples are aligned in two modal-specific spaces under the guidance of ID loss and CR loss.
• Figure 3: Effect of different numbers of REMs (i.e.,$M$) in each REM-G on the CUHK-PEDES dataset.
• Figure 4: Comparison of heatmaps between the base model and Beat. The Base model often attends to a large area, while the BEAT model can accurately focus on pedestrian areas.
• Figure 5: Feature visualization of the base model and Beat via t-SNE van2008visualizing on CUHK-PEDES. We show the changing process of cross-modal feature distributions with training. The feature of each image and text is marked as a circle and a triangle, respectively. Each identity is indicated in a specific color. For clarity, we only visualize global visual and textual features.