Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Model

TL;DR

This work presents a multi-task dual-encoder framework that learns cross-lingual sentence representations by jointly training on monolingual tasks and a translation-bridge task to align embeddings across languages. The model uses a shared Transformer-based encoder with task-specific heads and demonstrates strong English task performance while achieving effective zero-shot cross-lingual transfer in retrieval, STS, NLI, and sentiment analysis. The results show that incorporating monolingual data and cross-lingual translation signals improves alignment and transfer, with detailed analyses of embedding-space structure. These findings suggest that cross-lingual multi-task learning can yield robust multilingual representations suitable for diverse downstream tasks.

Abstract

A significant roadblock in multilingual neural language modeling is the lack of labeled non-English data. One potential method for overcoming this issue is learning cross-lingual text representations that can be used to transfer the performance from training on English tasks to non-English tasks, despite little to no task-specific non-English data. In this paper, we explore a natural setup for learning cross-lingual sentence representations: the dual-encoder. We provide a comprehensive evaluation of our cross-lingual representations on a number of monolingual, cross-lingual, and zero-shot/few-shot learning tasks, and also give an analysis of different learned cross-lingual embedding spaces.

Figures (1)

• Figure 1: Multi-task dual-encoder model with native tasks and a bridging translation task. The terms PAR, INP, RES refer to parent, input, and response respectively. ENC refers to the shared encoder $g$, FC refers to fully connected layers, and DOT refers to dot product. Finally, FEATURE TRANSFORM refers to the feature vector used for natural language inference.