Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models

Mehar Bhatia, Sahithya Ravi, Aditya Chinchure, Eunjeong Hwang, Vered Shwartz

TL;DR

GlobalRG tackles the multicultural limitations of vision-language models by introducing two challenging tasks: Retrieval across Universals and Cultural Visual Grounding, spanning 50 cultures for universal concepts and 15 cultures for grounding across 8 regions. The benchmark couples a novel diversity@k metric with standard relevance measures to reveal systemic Western biases in both retrieval and grounding across diverse architectures. Across extensive evaluations, results show meaningful cultural disparities and biases tied to training data and objectives, underscoring the need for culturally diverse pretraining data and targeted objectives. The work highlights practical implications for fair and inclusive cross-cultural AI systems and points toward data collection and objective design as key levers for improvement.

Abstract

Despite recent advancements in vision-language models, their performance remains suboptimal on images from non-western cultures due to underrepresentation in training datasets. Various benchmarks have been proposed to test models' cultural inclusivity, but they have limited coverage of cultures and do not adequately assess cultural diversity across universal as well as culture-specific local concepts. To address these limitations, we introduce the GlobalRG benchmark, comprising two challenging tasks: retrieval across universals and cultural visual grounding. The former task entails retrieving culturally diverse images for universal concepts from 50 countries, while the latter aims at grounding culture-specific concepts within images from 15 countries. Our evaluation across a wide range of models reveals that the performance varies significantly across cultures -- underscoring the necessity for enhancing multicultural understanding in vision-language models.

From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models

TL;DR

GlobalRG tackles the multicultural limitations of vision-language models by introducing two challenging tasks: Retrieval across Universals and Cultural Visual Grounding, spanning 50 cultures for universal concepts and 15 cultures for grounding across 8 regions. The benchmark couples a novel diversity@k metric with standard relevance measures to reveal systemic Western biases in both retrieval and grounding across diverse architectures. Across extensive evaluations, results show meaningful cultural disparities and biases tied to training data and objectives, underscoring the need for culturally diverse pretraining data and targeted objectives. The work highlights practical implications for fair and inclusive cross-cultural AI systems and points toward data collection and objective design as key levers for improvement.

Abstract

Despite recent advancements in vision-language models, their performance remains suboptimal on images from non-western cultures due to underrepresentation in training datasets. Various benchmarks have been proposed to test models' cultural inclusivity, but they have limited coverage of cultures and do not adequately assess cultural diversity across universal as well as culture-specific local concepts. To address these limitations, we introduce the GlobalRG benchmark, comprising two challenging tasks: retrieval across universals and cultural visual grounding. The former task entails retrieving culturally diverse images for universal concepts from 50 countries, while the latter aims at grounding culture-specific concepts within images from 15 countries. Our evaluation across a wide range of models reveals that the performance varies significantly across cultures -- underscoring the necessity for enhancing multicultural understanding in vision-language models.
Paper Structure (41 sections, 1 equation, 5 figures, 12 tables)

This paper contains 41 sections, 1 equation, 5 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Geo-Diversity Problem.
  4. Geo-Diverse Benchmarks.
  5. Task 1: Retrieval across Universals
  6. Dataset Collection
  7. Textual Queries.
  8. Images.
  9. Task Definition and Evaluation Setup
  10. Models
  11. Results and Analysis
  12. RQ$_1$: Are VLMs able to retrieve relevant and culturally diverse images for universal concept words?
  13. RQ$_2$: Do VLMs exhibit biases towards images from specific cultures?
  14. RQ$_3$: What are the challenges faced by VLMs in achieving high cultural diversity?
  15. Task 2: Cultural Visual Grounding
  16. ...and 26 more sections

Figures (5)

  • Figure 1: An example instance from each task in GlobalRG: i) Retrieval Across Universals measures the ability of VLMs to retrieve culturally diverse images for a query q. ii) Cultural Visual Grounding aims to evaluate the ability of VLMs to identify a cultural concept q.
  • Figure 2: Top 5 images retrieved for a sample of the universals by models CLIP, CoCA and BLIP-2. Each image is annotated with a flag representing the country, and the background colour of the flag represents the region.
  • Figure 3: Country-level Accuracy of each model on the Cultural Visual Grounding task.
  • Figure 4: Culture group-level Accuracy for Cultural Visual Grounding.
  • Figure 5: Qualitative Examples showing the performance of specialist and generalist models on Cultural Visual Grounding task.