WorldValuesBench: A Large-Scale Benchmark Dataset for Multi-Cultural Value Awareness of Language Models

TL;DR

WorldValuesBench (WVB) provides a globally diverse, large-scale benchmark for multi-cultural value awareness by transforming World Values Survey data into over 20 million (demographic attributes, value question) → answer instances. The dataset enables evaluation of how well language models predict demographically conditioned value ratings, using Wasserstein-1 distance to compare model and human distributions. A case study with a WVB-probe (36 questions, 3 demographics, 8,280 examples) shows substantial room for improvement across both open and closed models, with demographic conditioning yielding mixed results. This work offers a new research direction for safe, personalized LM outputs that respect cultural context while highlighting limitations and guiding future improvements in demography-aware alignment.

Abstract

The awareness of multi-cultural human values is critical to the ability of language models (LMs) to generate safe and personalized responses. However, this awareness of LMs has been insufficiently studied, since the computer science community lacks access to the large-scale real-world data about multi-cultural values. In this paper, we present WorldValuesBench, a globally diverse, large-scale benchmark dataset for the multi-cultural value prediction task, which requires a model to generate a rating response to a value question based on demographic contexts. Our dataset is derived from an influential social science project, World Values Survey (WVS), that has collected answers to hundreds of value questions (e.g., social, economic, ethical) from 94,728 participants worldwide. We have constructed more than 20 million examples of the type "(demographic attributes, value question) $\rightarrow$ answer" from the WVS responses. We perform a case study using our dataset and show that the task is challenging for strong open and closed-source models. On merely $11.1\%$, $25.0\%$, $72.2\%$, and $75.0\%$ of the questions, Alpaca-7B, Vicuna-7B-v1.5, Mixtral-8x7B-Instruct-v0.1, and GPT-3.5 Turbo can respectively achieve $<0.2$ Wasserstein 1-distance from the human normalized answer distributions. WorldValuesBench opens up new research avenues in studying limitations and opportunities in multi-cultural value awareness of LMs.

Figures (10)

• Figure 1: Human values often depend on cultural contexts, such as, country, residential area, and education. Given a value question and demographic attributes, we examine if a language model exhibits awareness of the human answer distribution.
• Figure 2: Adapting the WVS Codebook (left, PDF) for computational modeling (right, JSON). For each value question, we convert its title, description, and answer choices into a single-sentence question that elicits a rating answer and can be included in an LM prompt.
• Figure 3: Percentage of questions where the Wasserstein 1-distance between the human and model distributions is less than a series of thresholds between 0 and 0.4 with step 0.05. In the three plots, the distributions are respectively obtained for all examples, the examples corresponding to participants from urban areas, and those corresponding to participants from rural residential areas in the WVB-probe. Each model is prompted without (dashed line) and with (solid line) demographic attributes.
• Figure 4: Human and model answer distributions for value questions Q1 and Q106. All participants in the WVB-probe set are considered.
• Figure 5: The prompt for Alpaca (7B) Instruct (46.7B) with demography attributes.