Multilingual Distributed Representations without Word Alignment

TL;DR

The paper presents a method for learning cross-lingual, sentence-level distributed representations without word alignments by training a bilingual compositional vector model (biCVM) with a simple additive composition and a contrastive bilingual objective. Using parallel corpora, the model learns shared semantic representations across languages, achieving state-of-the-art results on cross-lingual document classification and demonstrating cross-language semantic relationships via pivot-language data. The approach is data-efficient, flexible in training data form, and extensible to multiple languages and richer compositional models. It offers practical impact for multilingual NLP tasks and low-resource language transfer, with potential applications in machine translation and multilingual information extraction.

Abstract

Distributed representations of meaning are a natural way to encode covariance relationships between words and phrases in NLP. By overcoming data sparsity problems, as well as providing information about semantic relatedness which is not available in discrete representations, distributed representations have proven useful in many NLP tasks. Recent work has shown how compositional semantic representations can successfully be applied to a number of monolingual applications such as sentiment analysis. At the same time, there has been some initial success in work on learning shared word-level representations across languages. We combine these two approaches by proposing a method for learning distributed representations in a multilingual setup. Our model learns to assign similar embeddings to aligned sentences and dissimilar ones to sentence which are not aligned while not requiring word alignments. We show that our representations are semantically informative and apply them to a cross-lingual document classification task where we outperform the previous state of the art. Further, by employing parallel corpora of multiple language pairs we find that our model learns representations that capture semantic relationships across languages for which no parallel data was used.

Figures (3)

• Figure 1: Description of a bilingual model with parallel input sentences $a$ and $b$. The objective function of this model is to minimize the distance between the sentence level encoding of the bitext. Principally any composition function can be used to generate the compositional sentence level representations. The composition function is represented by the CVM boxes in the diagram above.
• Figure 2: Classification accuracy for a number of models (see Table \ref{['tab:results1k']} for model descriptions). The left chart shows results for these models when trained on English data and evaluated on German data, the right chart vice versa.
• Figure 3: The left scatter plot shows t-SNE projections for a weekdays in all three languages using the representations learned in the biCVM+ model. Even though the model did not use any parallel French-German data during training, it still learns semantic similarity between these two languages using English as a pivot. To highlight this, the right plot shows another set of words (months of the year) using only the German and French words.