Multilinguality as Sense Adaptation

TL;DR

SENSIA reframes multilingual transfer as sense adaptation, aligning latent sense mixtures and contextual representations across languages rather than relying on broad parameter sharing. It builds on the Backpack architecture, using a three-part loss—sense alignment, context alignment, and target-language language modeling—optimized through a three-phase curriculum to adapt English models to Estonian, Indonesian, Swahili, and Turkish with parallel data. Across semantic alignment metrics and downstream reasoning tasks, SENSIA demonstrates data-efficient, robust performance that often surpasses finetuning and dense-representation baselines while narrowing the gap to monolingual models; analyses show preservation of local sense topology and global structure relative to English via Procrustes mapping. The work emphasizes a linguistically motivated pathway for cross-lingual transfer, highlights data and tokenizer limitations, and outlines clear directions for scaling, non-Latin script coverage, and broader parallel-data supervision.

Abstract

We approach multilinguality as sense adaptation: aligning latent meaning representations across languages rather than relying solely on shared parameters and scale. In this paper, we introduce SENse-based Symmetric Interlingual Alignment (SENSIA), which adapts a Backpack language model from one language to another by explicitly aligning sense-level mixtures and contextual representations on parallel data, while jointly training a target-language language modeling loss to preserve fluency. Across benchmarks on four typologically diverse languages, SENSIA generally outperforms comparable multilingual alignment methods and achieves competitive accuracy against monolingual from-scratch baselines while using 2-4x less target-language data. Analyses of learned sense geometry indicate that local sense topology and global structure relative to English are largely preserved, and ablations show that the method is robust in terms of design and scale.

Figures (4)

• Figure 1: SENSIA overview. (A) A Backpack represents each input token as a context-dependent mixture of $K$ latent sense vectors. (B) The sense loss aligns sense-level representations of meaning. (C) The context loss aligns context-level representations via the last non-pad tokens. (D) A target-language language modeling loss preserves fluency while senses and contexts are aligned across languages.
• Figure 2: Training dynamics on the FLORES-200 validation set. (a) Mean target-side sense entropy decreases and stabilizes as senses specialize without collapsing. (b--c) src$\rightarrow$tgt and tgt$\rightarrow$src recall@1 rapidly rise and remain high, indicating a stable, aligned bilingual space during adaptation.
• Figure 3: Sense Topology. For each aligned word pair, we ask if its senses relate to each other similarly across languages, rather than requiring one-to-one matches between individual senses. We compute $K$ sense vectors for English and the target language, pooling if a word is split into subwords. We then build cosine gram matrices over senses in each language and compare their upper triangles with a Spearman correlation $\rho$. A high $\rho$ means that the pattern of inter-sense relationships is preserved even if the exact locations of individual senses differ, whereas exact one-to-one matches indicates simple memorization of the English space.
• Figure 4: Procrustes Analysis. Target-language sense embeddings may be rotated relative to English even if their structure is identical. We therefore learn an orthogonal transform $Q^{\star}$ that best aligns $T$ to $E$, and measure cosine similarity between the aligned spaces to assess how similar their global geometry is.