Uncovering inequalities in new knowledge learning by large language models across different languages

TL;DR

This work explores inequalities in new knowledge learning by LLMs across different languages and four key dimensions: effectiveness, transferability, prioritization, and robustness and analyzes the underlying causes from linguistic perspectives, pretraining characteristics, and tokenizer design.

Abstract

As large language models (LLMs) gradually become integral tools for problem solving in daily life worldwide, understanding linguistic inequality is becoming increasingly important. Existing research has primarily focused on static analyses that assess the disparities in the existing knowledge and capabilities of LLMs across languages. However, LLMs are continuously evolving, acquiring new knowledge to generate up-to-date, domain-specific responses. Investigating linguistic inequalities within this dynamic process is, therefore, also essential. In this paper, we explore inequalities in new knowledge learning by LLMs across different languages and four key dimensions: effectiveness, transferability, prioritization, and robustness. Through extensive experiments under two settings (in-context learning and fine-tuning) using both proprietary and open-source models, we demonstrate that low-resource languages consistently face disadvantages across all four dimensions. By shedding light on these disparities, we aim to raise awareness of linguistic inequalities in LLMs' new knowledge learning, fostering the development of more inclusive and equitable future LLMs.

Figures (6)

• Figure 1: a. LLMs struggle to provide current, relevant responses and to deliver precise, expert-level answers in specific domains. b. There are two techniques to enhance LLMs with new knowledge: in-context learning and fine-tuning. c. Four key inequalities emerge in new knowledge learning by LLMs across different languages.
• Figure 2: a. The performance of GPT-4o-mini in learning new knowledge. b. The performance of Llama-3.1-8B in learning new knowledge. Compared to high-resource languages, LLMs face greater challenges in learning new knowledge in low-resource languages in terms of both efficiency and accuracy.
• Figure 3: (a, c) The performance of GPT-4o-mini in transferring new knowledge under the fine-tuning setting and the underlying inequality. (b, d) The performance of GPT-4o-mini in transferring new knowledge under the in-context learning setting and the underlying inequality. New knowledge acquired by LLMs can be more easily transferred to high-resource languages than to low-resource languages.
• Figure 4: (a, d) Inequality in knowledge conflict scenarios. (b, c, e, f) Specific knowledge conflict scenarios for GPT-4o-mini and Llama-3.1-8B in both the fine-tuning and in-context learning settings. New knowledge in high-resource languages is often prioritized over that in low-resource languages.
• Figure 5: (a, c) The inequality in resisting errors in the fine-tuning setting. (b, d) The inequality in resisting errors in the in-context learning setting. LLMs tend to be more resistant to incorrect knowledge in high-resource languages than in low-resource languages.