CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer

TL;DR

This work addresses cross-lingual cross-modal retrieval without requiring annotated vision–target-language pairs by proposing CL2CM, a dual-network framework that leverages a cross-lingual network for multi-level alignment and a cross-modal network for vision–target-language matching. Cross-lingual knowledge transfer via relational distillation transfers CL-informed semantics to the CM stream, while self-supervised word-level alignment via Optimal Transport mitigates translation noise. The approach achieves strong results on multilingual image–text and video–text retrieval benchmarks (Multi30K, MSCOCO, VATEX) with competitive efficiency, demonstrating the practical viability of scalable, multilingual CCR. Overall, CL2CM provides a principled pathway to leverage multilingual pre-trained models for reliable cross-modal alignment without costly V–T annotations, enabling broad applicability in multilingual information access.

Abstract

Cross-lingual cross-modal retrieval has garnered increasing attention recently, which aims to achieve the alignment between vision and target language (V-T) without using any annotated V-T data pairs. Current methods employ machine translation (MT) to construct pseudo-parallel data pairs, which are then used to learn a multi-lingual and multi-modal embedding space that aligns visual and target-language representations. However, the large heterogeneous gap between vision and text, along with the noise present in target language translations, poses significant challenges in effectively aligning their representations. To address these challenges, we propose a general framework, Cross-Lingual to Cross-Modal (CL2CM), which improves the alignment between vision and target language using cross-lingual transfer. This approach allows us to fully leverage the merits of multi-lingual pre-trained models (e.g., mBERT) and the benefits of the same modality structure, i.e., smaller gap, to provide reliable and comprehensive semantic correspondence (knowledge) for the cross-modal network. We evaluate our proposed approach on two multilingual image-text datasets, Multi30K and MSCOCO, and one video-text dataset, VATEX. The results clearly demonstrate the effectiveness of our proposed method and its high potential for large-scale retrieval.

Figures (6)

• Figure 2: Overview of the proposed CL2CM framework. It consists of a cross-lingual (CL) with multi-level alignment and a cross-modal (CM) network with instance-level alignment. We aim to improve the alignment quality between vision and target-language using cross-lingual knowledge transfer.
• Figure 3: Visualization of the generated pseudo-label in self-supervised word-level alignment. The red box represents the incorrect translated word of the corresponding source-language word.
• Figure 4: An illustration of the CL knowledge transfer.
• Figure 5: Ablation study to investigate the impact of different CL knowledge transfer approaches on Multi30K. Two-stage CL2CM variants initially train a CL network, followed by knowledge transfer to the CM network. The results indicate that CL knowledge transfer achieves significant performance gains, and the multi-level alignment in stage 2 can further improve the target-language representation learning.
• Figure 6: TSNE visualization of 20 images and their corresponding 5 target language (de) sentence representations on Multi30K. Dots with the same color indicate representations belonging to the same class.