"A Woman is More Culturally Knowledgeable than A Man?": The Effect of Personas on Cultural Norm Interpretation in LLMs
Mahammed Kamruzzaman, Hieu Nguyen, Nazmul Hassan, Gene Louis Kim
TL;DR
The paper investigates whether sociodemographic personas alter LLMs' interpretation of cultural norms, challenging the assumption that norm acceptability is purely culturally anchored. Using two cultural-norm datasets (NORMAD and EtiCor), 36 personas across 12 groups, and four LLMs with three prompting templates, the authors quantify how persona assignment shifts accuracy and reveals biases. Key findings show that socially desirable personas often yield higher accuracy, biases persist across gender, race, and ability, and model choice critically shapes results, with EtiCor generally more affected than NORMAD; Kendall's $\tau$ tests reveal statistically significant differences in several comparisons ($p$-values$<$0.001 in many cases). The work highlights the practical implications for personalization in culturally diverse contexts, emphasizing the need to address bias and ensure fair, reliable norm interpretation when employing persona-based prompting in LLMs.
Abstract
As the deployment of large language models (LLMs) expands, there is an increasing demand for personalized LLMs. One method to personalize and guide the outputs of these models is by assigning a persona -- a role that describes the expected behavior of the LLM (e.g., a man, a woman, an engineer). This study investigates whether an LLM's understanding of social norms varies across assigned personas. Ideally, the perception of a social norm should remain consistent regardless of the persona, since acceptability of a social norm should be determined by the region the norm originates from, rather than by individual characteristics such as gender, body size, or race. A norm is universal within its cultural context. In our research, we tested 36 distinct personas from 12 sociodemographic categories (e.g., age, gender, beauty) across four different LLMs. We find that LLMs' cultural norm interpretation varies based on the persona used and the norm interpretation also varies within a sociodemographic category (e.g., a fat person and a thin person as in physical appearance group) where an LLM with the more socially desirable persona (e.g., a thin person) interprets social norms more accurately than with the less socially desirable persona (e.g., a fat person). We also discuss how different types of social biases may contribute to the results that we observe.