Do You Know About My Nation? Investigating Multilingual Language Models' Cultural Literacy Through Factual Knowledge

TL;DR

XNationQA introduces a large-scale, parallel multilingual benchmark to evaluate cultural literacy across nine nations in seven languages, addressing Western-centric biases in prior QA benchmarks. The authors mine country-specific entities, craft four domain templates per entity, and expand to seven languages via translation and back-translation validation, enabling cross-language evaluation. They benchmark eight instruction-tuned multilingual LLMs and develop Total Coverage ($TC^d$) and Smooth Coverage ($SC^d$) metrics to quantify cross-language transfer, revealing strong Western-language advantages, limited cross-language transfer for open-source models, and a decoupling between linguistic competence and cultural knowledge. GPT-4 consistently demonstrates the strongest cross-lingual literacy, while open-source models exhibit substantial transfer gaps, underscoring the need for culturally inclusive training and evaluation to achieve truly global multilingual literacy.

Abstract

Most multilingual question-answering benchmarks, while covering a diverse pool of languages, do not factor in regional diversity in the information they capture and tend to be Western-centric. This introduces a significant gap in fairly evaluating multilingual models' comprehension of factual information from diverse geographical locations. To address this, we introduce XNationQA for investigating the cultural literacy of multilingual LLMs. XNationQA encompasses a total of 49,280 questions on the geography, culture, and history of nine countries, presented in seven languages. We benchmark eight standard multilingual LLMs on XNationQA and evaluate them using two novel transference metrics. Our analyses uncover a considerable discrepancy in the models' accessibility to culturally specific facts across languages. Notably, we often find that a model demonstrates greater knowledge of cultural information in English than in the dominant language of the respective culture. The models exhibit better performance in Western languages, although this does not necessarily translate to being more literate for Western countries, which is counterintuitive. Furthermore, we observe that models have a very limited ability to transfer knowledge across languages, particularly evident in open-source models.

Figures (13)

• Figure 1: An example evaluating an LLM's cultural literacy for Spain, in both English and Spanish. The model answers the war question correctly in both English and Spanish but fails on the leader question in both. Its performance on the national park and monument questions is language-dependent, highlighting inconsistencies in its cultural literacy across languages and topics.
• Figure 2: A detailed breakdown of model accuracy across different languages for each of the nine countries in XNationQA (see Table \ref{['tab:langs_iso']} for ISO codes). Each radar plot represents a model, with axes for the nine countries and colored lines for the seven languages. The plots demonstrate the significant performance disparities across both nations and languages. For instance, most models show stronger performance in English (blue line) across all countries, while struggling with non-Western languages like Hindi (magenta line). The irregular shapes of the plots for models like Bloomz or LLaMA-2 highlight inconsistent cultural knowledge across different nations.
• Figure 3: Distribution of entities across the nine nations considered in our study.
• Figure 4: Heatmaps for pairwise $TC$ of all language pairs, for -- (A) Bloomz-7B1, (B)LLaMA-2-7B-Chat, (C)Mistral-7B-Instruct, (D)Meta-LLaMA-3-8B-Instruct, (E) LLaMA-2-13B-Chat, (F) 13-billion Aya, (G) GPT-4 and (H) Mixtral-8x7B.
• Figure 5: Accuracy of models in different languages for questions on the birth year of leaders of different nations (see Table \ref{['tab:langs_iso']} for ISO code).