World in a Frame: Understanding Culture Mixing as a New Challenge for Vision-Language Models

TL;DR

CultureMix reveals a fundamental gap in vision-language models: while LVLMs recognize single cultural cues well, they falter in culturally mixed scenes where background cues disproportionately steer predictions. The authors build a 23k synthetic culture-mixing, diffusion-generated dataset (plus 100 real-world images) and evaluate 10 LVLMs across four subtasks, finding systematic declines in accuracy and increases in uncertainty as cultural distance grows. They analyze the mechanisms behind these failures, showing that background distractors and high-resource-region biases dominate model behavior, and demonstrate that both training-free prompts and supervised finetuning can improve robustness, with SFT offering the strongest gains in complex settings. The work introduces CultureMix-Real for real-world validation and provides a practical path forward by highlighting effective mitigation strategies, underscoring the need for culture-mixing-focused objectives to ensure LVLMs operate reliably in multicultural environments.

Abstract

In a globalized world, cultural elements from diverse origins frequently appear together within a single visual scene. We refer to these as culture mixing scenarios, yet how Large Vision-Language Models (LVLMs) perceive them remains underexplored. We investigate culture mixing as a critical challenge for LVLMs and examine how current models behave when cultural items from multiple regions appear together. To systematically analyze these behaviors, we construct CultureMix, a food Visual Question Answering (VQA) benchmark with 23k diffusion-generated, human-verified culture mixing images across four subtasks: (1) food-only, (2) food+food, (3) food+background, and (4) food+food+background. Evaluating 10 LVLMs, we find consistent failures to preserve individual cultural identities in mixed settings. Models show strong background reliance, with accuracy dropping 14% when cultural backgrounds are added to food-only baselines, and they produce inconsistent predictions for identical foods across different contexts. To address these limitations, we explore three robustness strategies. We find supervised fine-tuning using a diverse culture mixing dataset substantially improve model consistency and reduce background sensitivity. We call for increased attention to culture mixing scenarios as a critical step toward developing LVLMs capable of operating reliably in culturally diverse real-world environments.

Figures (19)

• Figure 1: Conceptual illustration of LVLMs in Culture Mixing scenarios. Real-world contexts often contain multiple cultural elements that humans can easily identify, yet LVLMs struggle to identify them.
• Figure 2: Dataset construction and evaluation pipeline.CultureMix aims to benchmark state-of-the-art LVLMs on their cultural knowledge in diverse mixing scenarios by asking models to identify food names and their countries of origin, featuring various combinations of foods and backgrounds from over 30 countries. All images are synthetically generated using diffusion models with human-in-the-loop validation. Model responses in this figure are from InternVL2-14B.
• Figure 3: Overall model performance on country and food name identification. (a) Accuracy comparison across models for each subtask. (b) Country identification target-prediction heatmaps for each subtask. For every golden country, the plots show the distribution of predicted countries, illustrating both correct predictions and systematic confusions across models.
• Figure 4: Effect of cultural distractors on country prediction label shifts.
• Figure 5: Effect of cultural distance between target food and background distractor in SFB on country identification accuracy and prediction entropy.