Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMs

TL;DR

This work questions the validity of closed-style MCQ surveys for evaluating LLM cultural alignment, demonstrating that alignment is highly sensitive to prompt constraint, option order, and language. By applying four probing methods and Anthropological Prompting to World Values Survey and Hofstede data across Bangladesh, Germany, USA, and the Philippines, the study shows unconstrained prompts yield richer, often more culturally aligned responses than forced MCQs, while exposing systematic biases in low-resource languages. The authors quantify alignment via hard and soft WVS metrics and Spearman correlations for Hofstede, revealing that unconstrained settings produce more robust, statistically significant cross-cultural signals, albeit with language-resource caveats. They advocate shifting toward flexible evaluation frameworks that rely on cultural proxies and curated data (e.g., CultureLLM) to better capture localized alignment and reduce Western-centric bias in LLMs. The findings have practical implications for designing culturally aware AI systems and for benchmarking LLMs in real-world, diverse user contexts.

Abstract

A large number of studies rely on closed-style multiple-choice surveys to evaluate cultural alignment in Large Language Models (LLMs). In this work, we challenge this constrained evaluation paradigm and explore more realistic, unconstrained approaches. Using the World Values Survey (WVS) and Hofstede Cultural Dimensions as case studies, we demonstrate that LLMs exhibit stronger cultural alignment in less constrained settings, where responses are not forced. Additionally, we show that even minor changes, such as reordering survey choices, lead to inconsistent outputs, exposing the limitations of closed-style evaluations. Our findings advocate for more robust and flexible evaluation frameworks that focus on specific cultural proxies, encouraging more nuanced and accurate assessments of cultural alignment in LLMs.

Figures (10)

• Figure 1: Responses of GPT-4o to a proposition from the World Values Survey (WVS) under varying levels of constraint. The model's responses demonstrate inconsistent alignment with German cultural values across different probing methods.
• Figure 2: Four probing methods used in the study, along with the Anthropological prompting. Language models are prompted in both English and the native languages of the cultures being studied.
• Figure 3: Projection of Llama 3.3 relative to the original country positions ($\bullet$) on the Inglehart–Welzel World Cultural Map (2023). Results are shown for four probing methods: $\blacksquare$FC (Forced Closed-Style), $\blacktriangle$FR (Forced Reverse Order), $\blacklozenge$FO (Forced Open-Ended), and $\bigstar$FU (Fully Unconstrained). The cultural map is redrawn using factor loadings from the WVS Survey Findings, with model projections overlaid. Unconstrained probing ($\bigstar$, $\blacklozenge$) yields positions closest to the original country locations, indicating stronger cultural alignment. Not all models are included to avoid congestion in the figure.
• Figure 4: Comparison of cross-cultural correlation per value across the four probing methods for all models. Color intensity encodes the correlation strength (deep blue = strong positive, deep red= strong negative, white = near-zero). Values closer to 1 indicate stronger alignment with expected cultural dimensions, and starred values (striped cells) highlight the values with statistical significance. Notably, the unconstrained settings achieve the highest proportion of statistically significant positive correlations, highlighting its stronger alignment performance across models and dimensions.
• Figure 5: Anthropological Prompting.