Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VULCA-Bench: A Multicultural Vision-Language Benchmark for Evaluating Cultural Understanding

Haorui Yu, Ramon Ruiz-Dolz, Diji Yang, Hang He, Fengrui Zhang, Qiufeng Yi

TL;DR

Vulca-Bench introduces a multicultural art critique benchmark to evaluate Vision-Language Models on hierarchical cultural understanding beyond surface perception. It operationalizes a five-layer framework (L1–L5) across 8 traditions using 225 culture-specific dimensions and 7,410 image-critique pairs, with bilingual Chinese–English critiques and a Cultural Symmetry Principle to ensure parity. Pilot evaluations with multiple VLMs reveal a substantial depth gap: L3–L5 reasoning is consistently harder than L1–L2, underscoring the need for architecture and prompting strategies that target higher-order cultural interpretation. The dataset, evaluation tools, and expert critiques are released under CC BY 4.0, enabling diagnostics, interpretability research, and culturally grounded model development, while recognizing limitations such as dataset distribution and linguistic scope.

Abstract

We introduce VULCA-Bench, a multicultural art-critique benchmark for evaluating Vision-Language Models' (VLMs) cultural understanding beyond surface-level visual perception. Existing VLM benchmarks predominantly measure L1-L2 capabilities (object recognition, scene description, and factual question answering) while under-evaluate higher-order cultural interpretation. VULCA-Bench contains 7,410 matched image-critique pairs spanning eight cultural traditions, with Chinese-English bilingual coverage. We operationalise cultural understanding using a five-layer framework (L1-L5, from Visual Perception to Philosophical Aesthetics), instantiated as 225 culture-specific dimensions and supported by expert-written bilingual critiques. Our pilot results indicate that higher-layer reasoning (L3-L5) is consistently more challenging than visual and technical analysis (L1-L2). The dataset, evaluation scripts, and annotation tools are available under CC BY 4.0 in the supplementary materials.

VULCA-Bench: A Multicultural Vision-Language Benchmark for Evaluating Cultural Understanding

TL;DR

Vulca-Bench introduces a multicultural art critique benchmark to evaluate Vision-Language Models on hierarchical cultural understanding beyond surface perception. It operationalizes a five-layer framework (L1–L5) across 8 traditions using 225 culture-specific dimensions and 7,410 image-critique pairs, with bilingual Chinese–English critiques and a Cultural Symmetry Principle to ensure parity. Pilot evaluations with multiple VLMs reveal a substantial depth gap: L3–L5 reasoning is consistently harder than L1–L2, underscoring the need for architecture and prompting strategies that target higher-order cultural interpretation. The dataset, evaluation tools, and expert critiques are released under CC BY 4.0, enabling diagnostics, interpretability research, and culturally grounded model development, while recognizing limitations such as dataset distribution and linguistic scope.

Abstract

We introduce VULCA-Bench, a multicultural art-critique benchmark for evaluating Vision-Language Models' (VLMs) cultural understanding beyond surface-level visual perception. Existing VLM benchmarks predominantly measure L1-L2 capabilities (object recognition, scene description, and factual question answering) while under-evaluate higher-order cultural interpretation. VULCA-Bench contains 7,410 matched image-critique pairs spanning eight cultural traditions, with Chinese-English bilingual coverage. We operationalise cultural understanding using a five-layer framework (L1-L5, from Visual Perception to Philosophical Aesthetics), instantiated as 225 culture-specific dimensions and supported by expert-written bilingual critiques. Our pilot results indicate that higher-layer reasoning (L3-L5) is consistently more challenging than visual and technical analysis (L1-L2). The dataset, evaluation scripts, and annotation tools are available under CC BY 4.0 in the supplementary materials.
Paper Structure (40 sections, 1 equation, 3 figures, 11 tables)

This paper contains 40 sections, 1 equation, 3 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset Construction
  4. Cultural Symmetry Principle
  5. Cultural Understanding Framework
  6. Data Sources
  7. Bilingual Critique Annotation
  8. Quality Assurance
  9. Dataset Statistics
  10. Data Quality Validation
  11. Pilot Evaluation
  12. Setup
  13. Dimension Coverage Results
  14. Implications
  15. Error Analysis: VLM Failures on L3--L5
  16. ...and 25 more sections

Figures (3)

  • Figure 1: Cross-cultural Case Gallery from Vulca-Bench (All 8 Traditions). Each artwork represents the highest L3--L5 dimension coverage for its cultural tradition: Chinese (16), Western (17), Japanese (14), Korean (13), Islamic (16), Indian (16), Mural (14), Hermitage (16). The full corpus contains 7,410 image--critique pairs across 8 traditions with 225 culture-specific dimensions.
  • Figure 2: Layer-wise dimension coverage. L1--L2 achieve $\geq$94%; L3--L5 decline to 72--89%, reflecting interpretive complexity.
  • Figure 3: Dataset Distribution. Culture distribution across 8 traditions (7,410 pairs). Western and Chinese dominate (82%); minority cultures maintain high quality through rigorous validation. Complete dataset is available under CC BY 4.0 in supplementary materials.