Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Measuring and Modeling "Culture" in LLMs: A Survey

Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Singh, Alham Fikri Aji, Jacki O'Neill, Ashutosh Modi, Monojit Choudhury

TL;DR

This survey addresses how culture is measured and modeled in LLMs by surveying over 90 papers and organizing them around demographic and semantic proxies, as well as probing methodologies. It reveals a heavy reliance on datasets as proxies and a predominance of black-box probing, with clear gaps including explicit culture definitions, broader semantic domains, robustness, multilingual datasets, and situated real-world studies. The work argues for explicit definitions of culture, expanded proxies, and more interpretable, diverse probing approaches to advance culturally inclusive LLMs. It also highlights the practical importance of understanding cultural biases for deploying LLMs across diverse populations and contexts. The study ultimately advocates a more interdisciplinary, situated research program to capture the thick descriptions of culture in NLP systems and their applications.

Abstract

We present a survey of more than 90 recent papers that aim to study cultural representation and inclusion in large language models (LLMs). We observe that none of the studies explicitly define "culture, which is a complex, multifaceted concept; instead, they probe the models on some specially designed datasets which represent certain aspects of "culture". We call these aspects the proxies of culture, and organize them across two dimensions of demographic and semantic proxies. We also categorize the probing methods employed. Our analysis indicates that only certain aspects of ``culture,'' such as values and objectives, have been studied, leaving several other interesting and important facets, especially the multitude of semantic domains (Thompson et al., 2020) and aboutness (Hershcovich et al., 2022), unexplored. Two other crucial gaps are the lack of robustness of probing techniques and situated studies on the impact of cultural mis- and under-representation in LLM-based applications.

Towards Measuring and Modeling "Culture" in LLMs: A Survey

TL;DR

This survey addresses how culture is measured and modeled in LLMs by surveying over 90 papers and organizing them around demographic and semantic proxies, as well as probing methodologies. It reveals a heavy reliance on datasets as proxies and a predominance of black-box probing, with clear gaps including explicit culture definitions, broader semantic domains, robustness, multilingual datasets, and situated real-world studies. The work argues for explicit definitions of culture, expanded proxies, and more interpretable, diverse probing approaches to advance culturally inclusive LLMs. It also highlights the practical importance of understanding cultural biases for deploying LLMs across diverse populations and contexts. The study ultimately advocates a more interdisciplinary, situated research program to capture the thick descriptions of culture in NLP systems and their applications.

Abstract

We present a survey of more than 90 recent papers that aim to study cultural representation and inclusion in large language models (LLMs). We observe that none of the studies explicitly define "culture, which is a complex, multifaceted concept; instead, they probe the models on some specially designed datasets which represent certain aspects of "culture". We call these aspects the proxies of culture, and organize them across two dimensions of demographic and semantic proxies. We also categorize the probing methods employed. Our analysis indicates that only certain aspects of ``culture,'' such as values and objectives, have been studied, leaving several other interesting and important facets, especially the multitude of semantic domains (Thompson et al., 2020) and aboutness (Hershcovich et al., 2022), unexplored. Two other crucial gaps are the lack of robustness of probing techniques and situated studies on the impact of cultural mis- and under-representation in LLM-based applications.
Paper Structure (15 sections, 3 figures)

This paper contains 15 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Culture in the Social Sciences
  3. Culture in NLP
  4. Method
  5. Searching Relevant Papers
  6. Taxonomy: Defining Culture
  7. Proxies of Culture
  8. Taxonomy: Probing Methods
  9. Findings: Defining Culture
  10. Demographic Proxies
  11. Semantic Proxies
  12. Findings: Probing Methods
  13. Gaps and Recommendations
  14. Conclusion
  15. Black Box Probing Methods

Figures (3)

  • Figure 1: Organizations of papers based on the "definition of culture."
  • Figure 2: Organization of papers based on the methods used.
  • Figure :