Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Aesthetics: Cultural Competence in Text-to-Image Models

Nithish Kannen, Arif Ahmad, Marco Andreetto, Vinodkumar Prabhakaran, Utsav Prabhu, Adji Bousso Dieng, Pushpak Bhattacharyya, Shachi Dave

TL;DR

This work introduces CUBE, a scalable benchmark for assessing cultural competence in text-to-image models along cultural awareness and cultural diversity. It constructs CUBE-CSpace (~300K artifacts) via a KB-augmented LLM pipeline over 8 geo-cultural countries and 3 concepts, and derives CUBE-1K (1000 prompts) for awareness testing. It also formalizes Cultural Diversity (CD) using Quality-Weighted Vendi Scores to jointly evaluate diversity and image quality across seeds and prompts. Human studies reveal substantial gaps in both awareness and diversity, motivating explicit cultural-competence objectives for future T2I models and providing a foundation for broader, globally inclusive multimodal AI evaluation. The authors share datasets and code publicly to encourage adoption and extension by the community.

Abstract

Text-to-Image (T2I) models are being increasingly adopted in diverse global communities where they create visual representations of their unique cultures. Current T2I benchmarks primarily focus on faithfulness, aesthetics, and realism of generated images, overlooking the critical dimension of cultural competence. In this work, we introduce a framework to evaluate cultural competence of T2I models along two crucial dimensions: cultural awareness and cultural diversity, and present a scalable approach using a combination of structured knowledge bases and large language models to build a large dataset of cultural artifacts to enable this evaluation. In particular, we apply this approach to build CUBE (CUltural BEnchmark for Text-to-Image models), a first-of-its-kind benchmark to evaluate cultural competence of T2I models. CUBE covers cultural artifacts associated with 8 countries across different geo-cultural regions and along 3 concepts: cuisine, landmarks, and art. CUBE consists of 1) CUBE-1K, a set of high-quality prompts that enable the evaluation of cultural awareness, and 2) CUBE-CSpace, a larger dataset of cultural artifacts that serves as grounding to evaluate cultural diversity. We also introduce cultural diversity as a novel T2I evaluation component, leveraging quality-weighted Vendi score. Our evaluations reveal significant gaps in the cultural awareness of existing models across countries and provide valuable insights into the cultural diversity of T2I outputs for under-specified prompts. Our methodology is extendable to other cultural regions and concepts, and can facilitate the development of T2I models that better cater to the global population.

Beyond Aesthetics: Cultural Competence in Text-to-Image Models

TL;DR

This work introduces CUBE, a scalable benchmark for assessing cultural competence in text-to-image models along cultural awareness and cultural diversity. It constructs CUBE-CSpace (~300K artifacts) via a KB-augmented LLM pipeline over 8 geo-cultural countries and 3 concepts, and derives CUBE-1K (1000 prompts) for awareness testing. It also formalizes Cultural Diversity (CD) using Quality-Weighted Vendi Scores to jointly evaluate diversity and image quality across seeds and prompts. Human studies reveal substantial gaps in both awareness and diversity, motivating explicit cultural-competence objectives for future T2I models and providing a foundation for broader, globally inclusive multimodal AI evaluation. The authors share datasets and code publicly to encourage adoption and extension by the community.

Abstract

Text-to-Image (T2I) models are being increasingly adopted in diverse global communities where they create visual representations of their unique cultures. Current T2I benchmarks primarily focus on faithfulness, aesthetics, and realism of generated images, overlooking the critical dimension of cultural competence. In this work, we introduce a framework to evaluate cultural competence of T2I models along two crucial dimensions: cultural awareness and cultural diversity, and present a scalable approach using a combination of structured knowledge bases and large language models to build a large dataset of cultural artifacts to enable this evaluation. In particular, we apply this approach to build CUBE (CUltural BEnchmark for Text-to-Image models), a first-of-its-kind benchmark to evaluate cultural competence of T2I models. CUBE covers cultural artifacts associated with 8 countries across different geo-cultural regions and along 3 concepts: cuisine, landmarks, and art. CUBE consists of 1) CUBE-1K, a set of high-quality prompts that enable the evaluation of cultural awareness, and 2) CUBE-CSpace, a larger dataset of cultural artifacts that serves as grounding to evaluate cultural diversity. We also introduce cultural diversity as a novel T2I evaluation component, leveraging quality-weighted Vendi score. Our evaluations reveal significant gaps in the cultural awareness of existing models across countries and provide valuable insights into the cultural diversity of T2I outputs for under-specified prompts. Our methodology is extendable to other cultural regions and concepts, and can facilitate the development of T2I models that better cater to the global population.
Paper Structure (59 sections, 6 equations, 10 figures, 14 tables)

This paper contains 59 sections, 6 equations, 10 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. T2I Evaluation.
  4. Culture in Language Technologies.
  5. Culture in Vision.
  6. Construction of CUBE
  7. CUBE-CSpace
  8. Refinement.
  9. CUBE-1K
  10. Evaluating Cultural Awareness
  11. Human Annotation
  12. Results
  13. Evaluating Cultural Diversity
  14. Cultural Diversity (CD)
  15. Foundation: Vendi Scores
  16. ...and 44 more sections

Figures (10)

  • Figure 1: Images from a SOTA T2I model demonstrating its lack of cultural diversity: (a) and (b) and cultural awareness: (c) and (d). (a) Images for "High definition photo of a monument" lack architectural and global diversity. (b) Images for "Image of Nigerian dish" lack the rich diversity in Nigerian cuisine. (c) "Image of Jagannath Temple from India" produces an incorrect depiction of the temple. (d) "Image of Japanese dish Kabayaki" produces an incorrect and cartoonized photo.
  • Figure 2: Framework for evaluating cultural competence in T2I models. The top subfigure shows the definition of cultural concepts and the extraction of concept space from KB + LLM. The bottom shows example task prompts to probe the model for cultural awareness and cultural diversity.
  • Figure 3: Examples of human evaluation results on cultural awareness for T2I models with high and low scores on faithfulness and realism. More qualitative examples are in Figures \ref{['fig:qualitative_examples-1']} and \ref{['fig:qualitative_examples-2']}.
  • Figure 4: Human annotation interface. Each question was annotated by 3 raters. The first question tested cultural relevance and the second and third question were only shown if the raters agreed the images had relevance to their cultures (yes/maybe). An additional text box was provided for raters to comment on unrealistic elements in the image.
  • Figure 5: Using within culture prompts, the above plot shows HPSv2 scores across all the three concepts to show quality of images produced for each geo-culture. Each subfigure compares the HPSv2 score for the models: (a) Imagen, and (b) SDXL
  • ...and 5 more figures

Theorems & Definitions (1)

  • Definition 5.1: Vendi Scores