Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-Lingual Representation Alignment Through Contrastive Image-Caption Tuning

Nathaniel Krasner, Nicholas Lanuzo, Antonios Anastasopoulos

TL;DR

The paper addresses cross-lingual NLU for low-resource languages by grounding multilingual text representations in a shared visual modality, avoiding parallel corpora. It introduces a joint text-vision contrastive framework using XLM-Roberta-Large and ViT-Base with 512-dim projections and a learnable temperature, trained on image-caption pairs where captions are translated into multiple languages. Results show that multilingual image-caption alignment can implicitly align text across languages, including Quechua, and supports cross-lingual NLI and bitext retrieval, with Quechua benefits evident when added to the training mix. Although not state-of-the-art, the approach provides a scalable bootstrapping path to improve cross-lingual capabilities for underserved languages and can help bootstrap higher-quality bilingual resources without parallel data.

Abstract

Multilingual alignment of sentence representations has mostly required bitexts to bridge the gap between languages. We investigate whether visual information can bridge this gap instead. Image caption datasets are very easy to create without requiring multilingual expertise, so this offers a more efficient alternative for low-resource languages. We find that multilingual image-caption alignment can implicitly align the text representations between languages, languages unseen by the encoder in pretraining can be incorporated into this alignment post-hoc, and these aligned representations are usable for cross-lingual Natural Language Understanding (NLU) and bitext retrieval.

Cross-Lingual Representation Alignment Through Contrastive Image-Caption Tuning

TL;DR

The paper addresses cross-lingual NLU for low-resource languages by grounding multilingual text representations in a shared visual modality, avoiding parallel corpora. It introduces a joint text-vision contrastive framework using XLM-Roberta-Large and ViT-Base with 512-dim projections and a learnable temperature, trained on image-caption pairs where captions are translated into multiple languages. Results show that multilingual image-caption alignment can implicitly align text across languages, including Quechua, and supports cross-lingual NLI and bitext retrieval, with Quechua benefits evident when added to the training mix. Although not state-of-the-art, the approach provides a scalable bootstrapping path to improve cross-lingual capabilities for underserved languages and can help bootstrap higher-quality bilingual resources without parallel data.

Abstract

Multilingual alignment of sentence representations has mostly required bitexts to bridge the gap between languages. We investigate whether visual information can bridge this gap instead. Image caption datasets are very easy to create without requiring multilingual expertise, so this offers a more efficient alternative for low-resource languages. We find that multilingual image-caption alignment can implicitly align the text representations between languages, languages unseen by the encoder in pretraining can be incorporated into this alignment post-hoc, and these aligned representations are usable for cross-lingual Natural Language Understanding (NLU) and bitext retrieval.
Paper Structure (13 sections, 2 equations, 1 figure, 3 tables)

This paper contains 13 sections, 2 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method, Experiments, and Results
  4. Experimental Setup
  5. Datasets
  6. Training
  7. Experiment 1: Does multilingual text-image alignment lead to text-text alignment?
  8. Experiment 2: Can a language unseen in the encoder's pretraining be added using only image caption tuning?
  9. Experiment 3: Are the downstream qualities of the representations preserved and is cross-lingual transfer possible?
  10. Conclusion
  11. Limitations
  12. Acknowledgements
  13. Complete XNLI Results

Figures (1)

  • Figure 1: t-SNE embeddings for the outputs of each encoder over flores-200 sentences. Translations are shown as cliques with lines connecting them. Visible lines, such as those in the two leftmost panels, indicate that representations of translated sentences are far from each other, ie., poor alignment. While not as clear as parallel text alignment (Eng-Pivot), multilingual image-text alignment (two rightmost panels) shows promising results.