Evaluating Large Language Models along Dimensions of Language Variation: A Systematik Invesdigatiom uv Cross-lingual Generalization

TL;DR

This work introduces a principled framework to evaluate large language models along dimensions of linguistic variation by modeling phonological, morphological, and lexical differences as Bayesian noise applied to a high-resource language neighbour. It defines three families of noisers, a composite-noise mechanism, and tractable posterior estimation to study cross-lingual degradation (PD) in zero-shot transfer across multiple tasks and languages. The authors demonstrate that PD trends in real closely-related languages align with those predicted by artificial languages, enabling cheap estimation of unseen-language performance and diagnosis of degradation sources. They also discuss error modes, realistic language generation for artificial languages, and avenues for mitigating PD through targeted preprocessing and resource-augmented strategies. Overall, the framework provides a scalable lens for understanding and addressing cross-lingual generalization gaps in LLMs.

Abstract

While large language models exhibit certain cross-lingual generalization capabilities, they suffer from performance degradation (PD) on unseen closely-related languages (CRLs) and dialects relative to their high-resource language neighbour (HRLN). However, we currently lack a fundamental understanding of what kinds of linguistic distances contribute to PD, and to what extent. Furthermore, studies of cross-lingual generalization are confounded by unknown quantities of CRL language traces in the training data, and by the frequent lack of availability of evaluation data in lower-resource related languages and dialects. To address these issues, we model phonological, morphological, and lexical distance as Bayesian noise processes to synthesize artificial languages that are controllably distant from the HRLN. We analyse PD as a function of underlying noise parameters, offering insights on model robustness to isolated and composed linguistic phenomena, and the impact of task and HRL characteristics on PD. We calculate parameter posteriors on real CRL-HRLN pair data and show that they follow computed trends of artificial languages, demonstrating the viability of our noisers. Our framework offers a cheap solution for estimating task performance on an unseen CRL given HRLN performance using its posteriors, as well as for diagnosing observed PD on a CRL in terms of its linguistic distances from its HRLN, and opens doors to principled methods of mitigating performance degradation.

Figures (7)

• Figure 1: Phonological/orthographic, morphological, and function and content word variation, and lexical choice difference, between hin and mai; $p^*$: bloomz7b1 MT output.
• Figure 2: PD% for each language, task, and noiser. $\psi^n$ depicts mean language PD trends. We show $(\theta_n, PD\%)$ points for real CRL-HRLN pairs using computed posteriors for X$\rightarrow$eng. See \ref{['sec:exp_setup']} for corresponding HRLNs per CRL. $\psi^{real}$ depicts trends for real CRLs, shown only when $\theta_{real}$ has a wide-enough range.
• Figure 3: Mean PD over all parametrizations per noiser for X$\rightarrow$eng
• Figure 4: Composing $\phi^{f,c}$ and $\phi^{m}$: studying $\psi^{f,c,m}$ given $\theta_m$ for Hindi for X$\rightarrow$eng. $\psi^{f,c}+\psi^{m}$ shows the theoretical additive trend.
• Figure 5: List of IPA character sets for the phonological noiser.