On The Conceptualization and Societal Impact of Cross-Cultural Bias

TL;DR

The paper analyzes 20 2025 works on cross-cultural bias in NLP to examine how bias is defined, who it harms, and how mitigation and adoption are assessed. It reveals pervasive inconsistencies in defining bias, stakeholder identification, and harm quantification, and notes that adoption impact is rarely evaluated. A five-dimension coding framework is proposed to standardize evaluation across concreteness of bias, stakeholders, harms, adoption effects, and mitigation. The study argues for explicit, participatory, and real-world impact assessments to improve the ethical deployment of culturally aware language technologies.

Abstract

Research has shown that while large language models (LLMs) can generate their responses based on cultural context, they are not perfect and tend to generalize across cultures. However, when evaluating the cultural bias of a language technology on any dataset, researchers may choose not to engage with stakeholders actually using that technology in real life, which evades the very fundamental problem they set out to address. Inspired by the work done by arXiv:2005.14050v2, I set out to analyse recent literature about identifying and evaluating cultural bias in Natural Language Processing (NLP). I picked out 20 papers published in 2025 about cultural bias and came up with a set of observations to allow NLP researchers in the future to conceptualize bias concretely and evaluate its harms effectively. My aim is to advocate for a robust assessment of the societal impact of language technologies exhibiting cross-cultural bias.