Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beneath the Surface of Consistency: Exploring Cross-lingual Knowledge Representation Sharing in LLMs

Maxim Ifergan, Leshem Choshen, Roee Aharoni, Idan Szpektor, Omri Abend

TL;DR

The paper tackles the problem that factual knowledge in LLMs varies across languages, exploring whether models store a single shared representation or language-specific copies. It introduces cross-lingual knowledge consistency (CKC) and cross-lingual knowledge representation sharing (CKR), and develops editing-based diagnostics to quantify CKR using the CLIKE dataset (35k facts, 13 languages, 7 scripts) across 7B-scale LLMs. Key findings show that high cross-language answer agreement (CKC) does not imply shared internal representations (CKR), with script similarity emerging as a dominant factor; fully shared knowledge could dramatically boost the best-language accuracy and cross-lingual performance. The work provides a methodology and dataset to guide the development of more robust, equitable multilingual LLMs, highlighting the need for approaches that promote true cross-lingual representation sharing beyond surface-level consistency.

Abstract

The veracity of a factoid is largely independent of the language it is written in. However, language models are inconsistent in their ability to answer the same factual question across languages. This raises questions about how LLMs represent a given fact across languages. We explore multilingual factual knowledge through two aspects: the model's ability to answer a query consistently across languages, and the ability to ''store'' answers in a shared representation for several languages. We propose a methodology to measure the extent of representation sharing across languages by repurposing knowledge editing methods. We examine LLMs with various multilingual configurations using a new multilingual dataset. We reveal that high consistency does not necessarily imply shared representation, particularly for languages with different scripts. Moreover, we find that script similarity is a dominant factor in representation sharing. Finally, we observe that if LLMs could fully share knowledge across languages, their accuracy in their best-performing language could benefit an increase of up to 150\% on average. These findings highlight the need for improved multilingual knowledge representation in LLMs and suggest a path for the development of more robust and consistent multilingual LLMs.

Beneath the Surface of Consistency: Exploring Cross-lingual Knowledge Representation Sharing in LLMs

TL;DR

The paper tackles the problem that factual knowledge in LLMs varies across languages, exploring whether models store a single shared representation or language-specific copies. It introduces cross-lingual knowledge consistency (CKC) and cross-lingual knowledge representation sharing (CKR), and develops editing-based diagnostics to quantify CKR using the CLIKE dataset (35k facts, 13 languages, 7 scripts) across 7B-scale LLMs. Key findings show that high cross-language answer agreement (CKC) does not imply shared internal representations (CKR), with script similarity emerging as a dominant factor; fully shared knowledge could dramatically boost the best-language accuracy and cross-lingual performance. The work provides a methodology and dataset to guide the development of more robust, equitable multilingual LLMs, highlighting the need for approaches that promote true cross-lingual representation sharing beyond surface-level consistency.

Abstract

The veracity of a factoid is largely independent of the language it is written in. However, language models are inconsistent in their ability to answer the same factual question across languages. This raises questions about how LLMs represent a given fact across languages. We explore multilingual factual knowledge through two aspects: the model's ability to answer a query consistently across languages, and the ability to ''store'' answers in a shared representation for several languages. We propose a methodology to measure the extent of representation sharing across languages by repurposing knowledge editing methods. We examine LLMs with various multilingual configurations using a new multilingual dataset. We reveal that high consistency does not necessarily imply shared representation, particularly for languages with different scripts. Moreover, we find that script similarity is a dominant factor in representation sharing. Finally, we observe that if LLMs could fully share knowledge across languages, their accuracy in their best-performing language could benefit an increase of up to 150\% on average. These findings highlight the need for improved multilingual knowledge representation in LLMs and suggest a path for the development of more robust and consistent multilingual LLMs.
Paper Structure (26 sections, 8 equations, 8 figures, 1 table)

This paper contains 26 sections, 8 equations, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Measuring CKC
  4. Measuring CKR
  5. Experimental Setup
  6. Data
  7. Fact Collection.
  8. Dataset Construction.
  9. Models
  10. Knowledge Editing Methods
  11. Metrics and Evaluation
  12. Results
  13. The Issue of Knowledge Variability
  14. Consistency Does Not Imply Representation Sharing
  15. The Key Role of the Language Script
  16. ...and 11 more sections

Figures (8)

  • Figure 1: Illustration of our method for distinguishing between cross-lingual consistency and representations sharing in a pairwise language setting. The sports (green) question demonstrates mere cross-lingual answer consistency, while the query about Allan Turing's birthplace (blue) exemplifies a shared underlying representation. Edits to the shared representation propagate across both languages, unlike the consistent-only fact. This method exposes the crucial difference between surface-level answer consistency and genuine cross-lingual knowledge sharing.
  • Figure 2: Cross-lingual performance variability: the accuracy of factual knowledge retrieval across different languages for several LLMs supporting different language sets. 'Any Language' (green) -- facts known in at least one language, 'Best Language' (orange) -- accuracy in the best-performing language, and 'Cross-lingual Average' -- mean accuracy across all 13 languages in the CLIKE dataset, with error bars indicating standard deviation.
  • Figure 3: Distribution and Expectation of CKC and CKR. Left: Percentage of facts known (NCL) or represented (NTL) across multiple languages for different models. Right: Expected number of languages per fact (E[NCL]) and expected number of languages sharing representation per edited fact (E[NTL]) for each model, illustrating the relationship between knowledge CKC and CKR.
  • Figure 4: The pairwise relationships of factual knowledge across languages for four language models, with all scores reported using Exact Match. $C(l_1,l_2)$ shows the percentage of facts known in the row language which were also retrieved in the column language. $SR(l_1,l_2)$ indicates the percentage of successfully edited facts in the row language which generalized to the column language using MEMIT. Under each language abbreviation is the overall accuracy for initial knowledge retrieval and the edition-reliability score for $C$ and $SR$ measures respectively. Languages are color-coded by script family.
  • Figure 5: Left: histogram of number of examples for each language, right: histogram of number of examples for each relation type.
  • ...and 3 more figures