Exploring Representational Disparities Between Multilingual and Bilingual Translation Models

TL;DR

This study addresses why complete parameter sharing in multilingual MT can underperform bilingual baselines for certain language pairs, especially in one-to-many translation. It analyzes representation geometry by measuring isotropy with $ID$ and $IsoScore$ on the same data across bilingual and multilingual models, revealing that multilingual decoders are consistently less isotropic and occupy fewer dimensions, while multilingual encoders show some capacity gains. The results suggest that language-specific information consumes much of the multilingual decoder space, with scale and multiparallel data modulating these effects and offering guidance for architecture design that balances sharing with language-specific capacity. Overall, the work provides a principled geometric lens on representational capacity, informing future multilingual modeling decisions to mitigate interference while preserving transfer benefits.

Abstract

Multilingual machine translation has proven immensely useful for both parameter efficiency and overall performance across many language pairs via complete multilingual parameter sharing. However, some language pairs in multilingual models can see worse performance than in bilingual models, especially in the one-to-many translation setting. Motivated by their empirical differences, we examine the geometric differences in representations from bilingual models versus those from one-to-many multilingual models. Specifically, we compute the isotropy of these representations using intrinsic dimensionality and IsoScore, in order to measure how the representations utilize the dimensions in their underlying vector space. Using the same evaluation data in both models, we find that for a given language pair, its multilingual model decoder representations are consistently less isotropic and occupy fewer dimensions than comparable bilingual model decoder representations. Additionally, we show that much of the anisotropy in multilingual decoder representations can be attributed to modeling language-specific information, therefore limiting remaining representational capacity.

Figures (7)

• Figure 1: Schematic of our hidden space utilization comparisons. We extract final layer representations from both a bilingual model and a multilingual model on the same set of parallel sentences. We compute the isotropy of these representations (Iso), and compare the two models.
• Figure 2: Depictions of 2D point clouds, their principal components, and their computed IsoScores and IDs. The left point cloud has high IsoScore due to even variance spread across principal components, but the right has lower IsoScore due to uneven variance spread. Both clouds have an ID of 1.0 as ID is less sensitive to variance spread.
• Figure 3: Semi-log plots of normalized singular values from SVD of bilingual decoder hidden states and multilingual decoder hidden states for the WMT-large en-{ru,zh} model. The spectra of bilingual decoder hidden states are better balanced than those of multilingual decoder hidden states. We use a semi-log scale for visibility.
• Figure 4: $\Delta$IsoScore values comparing the extent of the observed encoder isotropy increase ($Iso(X_{\text{enc}}^{\text{multi}}) - Iso(X_{\text{enc}}^{\text{bi}})$) to the extent of the observed isotropy decrease ($Iso(X_{\text{dec}}^{\text{bi}} )- Iso(X_{\text{dec}}^{\text{multi}})$) in our multilingual models, compared to their bilingual counterparts. Overall, the extent of the decoder isotropy decrease is larger than that of the encoder increase.
• Figure 5: $\Delta$IsoScore values between language-specific multilingual representations separated by language and overall multilingual representations, for both the encoder and decoder ($Iso(X^{\text{multi}}(s, t_k) - Iso(X^{\text{multi}}(s, \cup_j t_j)$). Large $\Delta$IsoScores between language-specific multilingual reps. and overall multilingual reps. indicate heavy encoding of language specificity in the decoder space.