Multilingual Text-to-Image Person Retrieval via Bidirectional Relation Reasoning and Aligning

TL;DR

This work addresses multilingual text-to-image person retrieval by jointly tackling data quality and cross-modal alignment across languages. It introduces LDAT to automatically build high-quality multilingual TIPR data via translation, noise filtering, and domain-specific rewriting, and Bi-IRRA to learn robust cross-language cross-modal representations. Bi-IRRA combines Bidirectional Implicit Relation Reasoning (Bi-IRR) with a Multi-dimensional Global Alignment (Md-GA) to model fine-grained local relations and coarse-grained global alignment, using cross-lingual D-MIM, bi-lingual MLM, ITC, and A-ITM pretexts. Extensive experiments on four multilingual TIPR benchmarks show state-of-the-art performance in both English and non-English settings, validating the effectiveness and practical impact of the proposed approach.

Abstract

Text-to-image person retrieval (TIPR) aims to identify the target person using textual descriptions, facing challenge in modality heterogeneity. Prior works have attempted to address it by developing cross-modal global or local alignment strategies. However, global methods typically overlook fine-grained cross-modal differences, whereas local methods require prior information to explore explicit part alignments. Additionally, current methods are English-centric, restricting their application in multilingual contexts. To alleviate these issues, we pioneer a multilingual TIPR task by developing a multilingual TIPR benchmark, for which we leverage large language models for initial translations and refine them by integrating domain-specific knowledge. Correspondingly, we propose Bi-IRRA: a Bidirectional Implicit Relation Reasoning and Aligning framework to learn alignment across languages and modalities. Within Bi-IRRA, a bidirectional implicit relation reasoning module enables bidirectional prediction of masked image and text, implicitly enhancing the modeling of local relations across languages and modalities, a multi-dimensional global alignment module is integrated to bridge the modality heterogeneity. The proposed method achieves new state-of-the-art results on all multilingual TIPR datasets. Data and code are presented in https://github.com/Flame-Chasers/Bi-IRRA.

Figures (11)

• Figure 1: Illustration of different TIPR methods. (a) Global-matching methods directly align global image and text feature representations. (b) Local-matching methods explicitly extract and align local image and text representations. (c) Our bidirectional implicit relation reasoning and aligning method not only implicitly reasons about the relations among all local tokens but also aligns global image and text representations in a multilingual environment.
• Figure 2: An Overview of LMs-driven Domain Adaptive Translation (LDAT) pipeline. It consists of three main phases: translation, filtering, and rewriting. In the translation phase, the source texts are translated into the initial target texts by an LLM. During the filtering phase, an evaluation metric is adopted to assess the noise level of each initial target text, based on which we divide the initial target texts into two groups: clean texts and noisy texts. These texts are then paired with their corresponding images and source texts, resulting in a clean dataset and a noisy dataset, respectively. The rewriting phase includes two steps: (1) finetuning an MLLM with the clean dataset, and (2) using the finetuned MLLM to rewrite the initial target texts from the noisy dataset, thereby producing a high-quality multilingual TIPR dataset.
• Figure 3: Overview of the proposed Bidirectional Implicit Relation Reasoning and Aligning (Bi-IRRA) framework. It consists of an image encoder, a text encoder, and a multimodal interaction encoder. With the triplet data as input, Bi-IRRA achieves cross-lingual cross-modal alignment through two key modules. The first is the Bidirectional Implicit Relation Reasoning (Bi-IRR) module, which has two pretext tasks— cross-lingual D-MIM and bi-lingual MLM. Bi-IRR facilitates the bidirectional modeling of fine-grained relations across languages and modalities. The second is the Multi-dimensional Global Alignment (Md-GA) module, which contains bi-lingual ITC and bi-lingual A-ITM pretext tasks to align global feature representations of texts and images.
• Figure 4: Illustration of bi-lingual MLM, as exemplified with English text-image pairs. It uses masked textual tokens as local fine-grained keys to align visual and textual information.
• Figure 5: Examples from CUHK-PEDES(M). Each image is paired with text descriptions in four different languages.