Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking Cross-lingual Gaps from a Statistical Viewpoint

Vihari Piratla, Purvam Jain, Darshan Singh, Partha Talukdar, Trevor Cohn

TL;DR

This work challenges the prevalent view that cross-lingual gaps in LLM recall arise primarily from misaligned parametric representations, proposing a bias-variance decomposition to explain source-to-target differences. By modeling source and target responses as stochastic logit distributions and introducing a mixing coefficient between high-variance and high-bias components, the authors show that most cross-lingual gaps are driven by target variance ($\kappa=1$) and that reducing variance via inference-time techniques—most notably ensembling and translation-based prompts—substantially closes the gap, improving target accuracy by $20$–$25\%$ across benchmarks. They validate the framework on ECLeKTic and MMLU (with mixup) across multiple models, finding that variance reduction consistently increases source-target agreement and that higher source confidence correlates with higher target confidence when biases are minimal. These findings suggest practical mitigation strategies focus on variance control at inference time rather than large-scale retraining, with broad implications for multilingual robustness and cross-lingual knowledge transfer in LLMs.

Abstract

Any piece of knowledge is usually expressed in one or a handful of natural languages on the web or in any large corpus. Large Language Models (LLMs) act as a bridge by acquiring knowledge from a source language and making it accessible when queried from target languages. Prior research has pointed to a cross-lingual gap, viz., a drop in accuracy when the knowledge is queried in a target language compared to when the query is in the source language. Existing research has rationalized divergence in latent representations in source and target languages as the source of cross-lingual gap. In this work, we take an alternative view and hypothesize that the variance of responses in the target language is the main cause of this gap. For the first time, we formalize the cross-lingual gap in terms of bias-variance decomposition. We present extensive experimental evidence which support proposed formulation and hypothesis. We then reinforce our hypothesis through multiple inference-time interventions that control the variance and reduce the cross-lingual gap. We demonstrate a simple prompt instruction to reduce the response variance, which improved target accuracy by 20-25% across different models.

Rethinking Cross-lingual Gaps from a Statistical Viewpoint

TL;DR

This work challenges the prevalent view that cross-lingual gaps in LLM recall arise primarily from misaligned parametric representations, proposing a bias-variance decomposition to explain source-to-target differences. By modeling source and target responses as stochastic logit distributions and introducing a mixing coefficient between high-variance and high-bias components, the authors show that most cross-lingual gaps are driven by target variance () and that reducing variance via inference-time techniques—most notably ensembling and translation-based prompts—substantially closes the gap, improving target accuracy by – across benchmarks. They validate the framework on ECLeKTic and MMLU (with mixup) across multiple models, finding that variance reduction consistently increases source-target agreement and that higher source confidence correlates with higher target confidence when biases are minimal. These findings suggest practical mitigation strategies focus on variance control at inference time rather than large-scale retraining, with broad implications for multilingual robustness and cross-lingual knowledge transfer in LLMs.

Abstract

Any piece of knowledge is usually expressed in one or a handful of natural languages on the web or in any large corpus. Large Language Models (LLMs) act as a bridge by acquiring knowledge from a source language and making it accessible when queried from target languages. Prior research has pointed to a cross-lingual gap, viz., a drop in accuracy when the knowledge is queried in a target language compared to when the query is in the source language. Existing research has rationalized divergence in latent representations in source and target languages as the source of cross-lingual gap. In this work, we take an alternative view and hypothesize that the variance of responses in the target language is the main cause of this gap. For the first time, we formalize the cross-lingual gap in terms of bias-variance decomposition. We present extensive experimental evidence which support proposed formulation and hypothesis. We then reinforce our hypothesis through multiple inference-time interventions that control the variance and reduce the cross-lingual gap. We demonstrate a simple prompt instruction to reduce the response variance, which improved target accuracy by 20-25% across different models.
Paper Structure (44 sections, 3 theorems, 19 equations, 15 figures, 3 tables)

This paper contains 44 sections, 3 theorems, 19 equations, 15 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Contributions:
  3. A Framework of Cross-lingual gaps
  4. Bias-variance decomposition of cross-lingual gaps
  5. Source-target gaps due to biases: knowledge did not transfer ($\kappa=0$).
  6. Source-target gaps due to variance: confidence did not transfer ($\kappa=1$).
  7. Decoding the nature of gaps.
  8. Further implications of unbiased noise in target
  9. Experiment Setup
  10. Experimental Results
  11. Cross-lingual gaps are due to variance
  12. Response Ensembling
  13. ECLeKTic.
  14. MMLU (with mixup).
  15. Observations.
  16. ...and 29 more sections

Key Result

Proposition 1

When knowledge did not transfer, the probability of shared response between source and target decreases with decreased response variance.

Figures (15)

  • Figure 1: If the model has knowledge barriers, we expect the responses in target biased as shown in left sketch of (a). In practice, we observe the target responses are distributed around source in such a way that their respective average values coincide as shown in the right sketch of (a). In (b), we show PCA projection of source and target responses where each response is a dot, and crosses represent centroid. We show KDE fitted distribution for three target languages for a numerical question in (c). Please refer Figure \ref{['fig:main_b_extended']}, \ref{['fig:main_c_extended']} of Appendix for additional plots, and Section \ref{['sec:intro']} for additional context.
  • Figure 2: Cross-lingual performance gaps on ECLeKTic and MMLU (with mixup) with different LLMs. Here, DS refers to the DeepSeek model. G-2.5-Flash, G-2.5-Pro are Flash and Pro flagship Gemini models from 2.5 class and GPT-5-mini & GPT-5 are Open AI models. All models exhibit a significant cross-lingual gap on the ECLeKTic benchmark. Please refer Section \ref{['sec:intro']} for additional context and Section \ref{['sec:expt_setup']} for dataset description.
  • Figure 3: Response ensembling from multiple forward passes gradually diminishes the source-target differences as illustrated on ECLeKTic (top) and MMLU (with mixup) (bottom). Oracle value shown in red in (a) is the best expected value. Each plot shows an estimated value of $\pi$. Please refer Section \ref{['sec:response_ensemble']} for details.
  • Figure 4: Prompt templates for two input ensembling approaches. k is the number of translations presented or generated. Please refer Section \ref{['sec:input_ensemble']}.
  • Figure 5: Cross-lingual gaps diminishes with reduced variance in source language. Answer confidence is defined in Section \ref{['sec:model:implications']}. High confidence in source leads to high confidence in target (Proposition \ref{['prop:src_tgt_conf_rel']}), which should lead to improved agreement if there is no source-to-target bias. Section \ref{['sec:confidence_vs_gaps']}.
  • ...and 10 more figures

Theorems & Definitions (6)

  • Proposition 1
  • Proposition 2
  • Proposition 3
  • proof
  • proof
  • proof