Multilingual-To-Multimodal (M2M): Unlocking New Languages with Monolingual Text

TL;DR

M2M presents a data- and parameter-efficient method to align multilingual text with frozen multimodal latent spaces using English supervision alone. By learning a small projection map with a pair of losses—alignment via MSE and a structure-preserving term—M2M enables cross-lingual retrieval and generation across image, audio, and cross-modal tasks without multilingual multimodal training data. The approach achieves strong zero-shot performance (e.g., Recall@10 on XTD) and demonstrates generalization to audio-text retrieval and cross-lingual text-to-image generation, with consistent qualitative evidence from alignment visualizations. While competitive, the method acknowledges gaps relative to state-of-the-art multilingual multimodal models and highlights avenues for token-level alignment and improved multilingual evaluation datasets.

Abstract

Multimodal models excel in English, supported by abundant image-text and audio-text data, but performance drops sharply for other languages due to limited multilingual multimodal resources. Existing solutions rely heavily on machine translation, while advances in multilingual text modeling remain underutilized. We introduce METAL, a lightweight alignment method that learns only a few linear layers using English text alone to map multilingual text embeddings into a multimodal space. Despite its simplicity, METAL matches baseline performance in English (94.9 percent Recall at 10) and achieves strong zero-shot transfer (89.5 percent Recall at 10 averaged across 11 languages, 10 unseen) on XTD text-to-image retrieval. Qualitative t-SNE visualizations show that multilingual embeddings align tightly with multimodal representations, while weight analysis reveals that the transformation reshapes embedding geometry rather than performing trivial rotations. Beyond image-text retrieval, METAL generalizes to audio-text retrieval and cross-lingual text-to-image generation. We release code and checkpoints at https://github.com/m2m-codebase/M2M , as well as multilingual evaluation datasets including MSCOCO Multilingual 30K (https://huggingface.co/datasets/piyushsinghpasi/mscoco-multilingual-30k ), AudioCaps Multilingual (https://huggingface.co/datasets/piyushsinghpasi/audiocaps-multilingual ), and Clotho Multilingual (https://huggingface.co/datasets/piyushsinghpasi/clotho-multilingual ), to facilitate further research.

Figures (10)

• Figure 1: Overview of M2M. Using only English text supervision, we learn a lightweight linear mapping that aligns multilingual text embeddings to a frozen multimodal text space (e.g., CLIP). English acts as a shared anchor during training, aligning multilingual text representations (triangles) to the multimodal text space (diamonds). This alignment implicitly transfers to other languages (stars and circles) without requiring any additional multilingual or multimodal supervision.
• Figure 2: Impact of $\lambda$ and $\beta$ on XTD image–text retrieval. Increasing $\lambda$ while reducing $\beta$ leads to consistent performance gains.
• Figure 3: Effect of scaling train data on XTD eval set for M2M-aligned model- Jina-CLIP-v1 $\times$ M-MPNET.
• Figure 4: t-SNE visualization (perplexity = 32) of text embeddings before and after alignment. Marker shapes denote visual clusters and colors indicate languages, with English J-CLIP text embeddings ($z_e$ or en_clip) in red. Before alignment (top), text embeddings ($z_m$ & $z_e$) are fragmented; after alignment (bottom), multilingual captions ($z_{m \to e}$) and J-CLIP text embeddings ($z_e$) align closely with shared visual clusters.
• Figure 5: Images generated by FLUX text-to-image model using the prompt "The city bus is traveling down the road" in multiple languages (non-English captions shown on images). Our M2M-aligned model produces similar quality images compared to baseline FLUX (both T5 and CLIP encoders), FLUX-T5 and FLUX-CLIP models.