Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-Cultural Expert-Level Art Critique Evaluation with Vision-Language Models

Haorui Yu, Ramon Ruiz-Dolz, Xuehang Wen, Fengrui Zhang, Qiufeng Yi

TL;DR

Key findings are that automated metrics are unreliable proxies for cultural depth, Western samples score higher than non-Western samples under the authors' sampling and rubric, and cross-judge scale mismatch makes naive score averaging unreliable, motivating a single primary judge with explicit calibration.

Abstract

Vision-Language Models (VLMs) excel at visual perception, yet their ability to interpret cultural meaning in art remains under-validated. We present a tri-tier evaluation framework for cross-cultural art-critique assessment: Tier I computes automated coverage and risk indicators offline; Tier II applies rubric-based scoring using a single primary judge across five dimensions; and Tier III calibrates the Tier II aggregate score to human ratings via isotonic regression, yielding a 5.2% reduction in MAE on a 152-sample held-out set. The framework outputs a calibrated cultural-understanding score for model selection and cultural-gap diagnosis, together with dimension-level diagnostics and risk indicators. We evaluate 15 VLMs on 294 expert anchors spanning six cultural traditions. Key findings are that (i) automated metrics are unreliable proxies for cultural depth, (ii) Western samples score higher than non-Western samples under our sampling and rubric, and (iii) cross-judge scale mismatch makes naive score averaging unreliable, motivating a single primary judge with explicit calibration. Dataset and code are available in the supplementary materials.

Cross-Cultural Expert-Level Art Critique Evaluation with Vision-Language Models

TL;DR

Key findings are that automated metrics are unreliable proxies for cultural depth, Western samples score higher than non-Western samples under the authors' sampling and rubric, and cross-judge scale mismatch makes naive score averaging unreliable, motivating a single primary judge with explicit calibration.

Abstract

Vision-Language Models (VLMs) excel at visual perception, yet their ability to interpret cultural meaning in art remains under-validated. We present a tri-tier evaluation framework for cross-cultural art-critique assessment: Tier I computes automated coverage and risk indicators offline; Tier II applies rubric-based scoring using a single primary judge across five dimensions; and Tier III calibrates the Tier II aggregate score to human ratings via isotonic regression, yielding a 5.2% reduction in MAE on a 152-sample held-out set. The framework outputs a calibrated cultural-understanding score for model selection and cultural-gap diagnosis, together with dimension-level diagnostics and risk indicators. We evaluate 15 VLMs on 294 expert anchors spanning six cultural traditions. Key findings are that (i) automated metrics are unreliable proxies for cultural depth, (ii) Western samples score higher than non-Western samples under our sampling and rubric, and (iii) cross-judge scale mismatch makes naive score averaging unreliable, motivating a single primary judge with explicit calibration. Dataset and code are available in the supplementary materials.
Paper Structure (68 sections, 6 equations, 11 figures, 19 tables)

This paper contains 68 sections, 6 equations, 11 figures, 19 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work
  4. VLM Evaluation Benchmarks
  5. Cultural AI and Bias Studies
  6. Computational Aesthetics
  7. LLM-as-Judge Paradigm
  8. Research Gap.
  9. Methodology
  10. Tri-Tier Framework
  11. Input/Output.
  12. Evaluation Modes.
  13. Output Metrics.
  14. Tier I: Automated Metrics.
  15. Tier II: Judge Scoring.
  16. ...and 53 more sections

Figures (11)

  • Figure 1: VLMs describe visual features (L1--L2) but miss cultural meaning (L3--L5). The expert critique explicitly covers symbolism (L3: pine = integrity), historical context (L4: Ni Zan, Yuan literati), and philosophy (L5: Daoist emptiness).
  • Figure 2: Vulca-Bench Evaluation Framework. Left: Three-tier pipeline (Tier I automated $\to$ Tier II judge $\to$ Tier III calibration). Right: L1--L5 cultural hierarchy. Solid arrows: deeper levels require LLM+human evaluation; dashed: keyword-based metrics may be adequate for coarse L1--L2 diagnostics. Output: $S_{\text{II}}^*$ = calibrated score.
  • Figure 4: Case Study. Left: high-quality critiques with L3--L5 cultural depth. Right: shallow outputs missing cultural interpretation despite fluent language. This illustrates why a risk-control layer is necessary: high Quality alone does not guarantee Accuracy or Alignment; Tier I flags such cases for $S_{\text{robust}}$ penalty.
  • Figure 5: Calibration reliability. Left: isotonic regression fit showing monotonic mapping from raw judge scores to human-aligned scores. Right: held-out validation ($n=152$) confirms 5.2% MAE improvement, with no systematic bias across score ranges.
  • Figure 6: Template risk detection. Samples in the upper-left quadrant (high Tier I, low Tier II) are flagged as "fluent-but-shallow" templates requiring $S_{\text{robust}}$ penalty. The 0.2 gap threshold (dashed line) achieves optimal F1=0.78.
  • ...and 6 more figures