Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Structured Framework for Evaluating and Enhancing Interpretive Capabilities of Multimodal LLMs in Culturally Situated Tasks

Haorui Yu, Ramon Ruiz-Dolz, Qiufeng Yi

TL;DR

This paper addresses the challenge of evaluating and enhancing multimodal language models in culturally embedded interpretive tasks, specifically Chinese art critique. It introduces VULCA, a three-part framework comprising a multi-dimensional human expert benchmark (MHEB) built from 163 expert commentaries, eight critic personas with a domain knowledge base, and a joint evaluation pipeline that merges vector-space semantic alignment with rubric-based scoring. Through cross-model experiments on models like Gemini, Qwen, and Llama, the study demonstrates substantial improvements in symbolic reasoning and argumentative coherence when persona guided prompting and domain grounding are applied, with reported gains of over 20% and 30% on selected dimensions for certain models. The framework and findings illuminate the importance of culturally grounded evaluation and intervention strategies for enabling epistemic alignment in complex, high-context domains, and the methodology generalizes to other humanities and social science subjects such as religion, history, and education.

Abstract

This study aims to test and evaluate the capabilities and characteristics of current mainstream Visual Language Models (VLMs) in generating critiques for traditional Chinese painting. To achieve this, we first developed a quantitative framework for Chinese painting critique. This framework was constructed by extracting multi-dimensional evaluative features covering evaluative stance, feature focus, and commentary quality from human expert critiques using a zero-shot classification model. Based on these features, several representative critic personas were defined and quantified. This framework was then employed to evaluate selected VLMs such as Llama, Qwen, or Gemini. The experimental design involved persona-guided prompting to assess the VLM's ability to generate critiques from diverse perspectives. Our findings reveal the current performance levels, strengths, and areas for improvement of VLMs in the domain of art critique, offering insights into their potential and limitations in complex semantic understanding and content generation tasks. The code used for our experiments can be publicly accessed at: https://github.com/yha9806/VULCA-EMNLP2025.

A Structured Framework for Evaluating and Enhancing Interpretive Capabilities of Multimodal LLMs in Culturally Situated Tasks

TL;DR

This paper addresses the challenge of evaluating and enhancing multimodal language models in culturally embedded interpretive tasks, specifically Chinese art critique. It introduces VULCA, a three-part framework comprising a multi-dimensional human expert benchmark (MHEB) built from 163 expert commentaries, eight critic personas with a domain knowledge base, and a joint evaluation pipeline that merges vector-space semantic alignment with rubric-based scoring. Through cross-model experiments on models like Gemini, Qwen, and Llama, the study demonstrates substantial improvements in symbolic reasoning and argumentative coherence when persona guided prompting and domain grounding are applied, with reported gains of over 20% and 30% on selected dimensions for certain models. The framework and findings illuminate the importance of culturally grounded evaluation and intervention strategies for enabling epistemic alignment in complex, high-context domains, and the methodology generalizes to other humanities and social science subjects such as religion, history, and education.

Abstract

This study aims to test and evaluate the capabilities and characteristics of current mainstream Visual Language Models (VLMs) in generating critiques for traditional Chinese painting. To achieve this, we first developed a quantitative framework for Chinese painting critique. This framework was constructed by extracting multi-dimensional evaluative features covering evaluative stance, feature focus, and commentary quality from human expert critiques using a zero-shot classification model. Based on these features, several representative critic personas were defined and quantified. This framework was then employed to evaluate selected VLMs such as Llama, Qwen, or Gemini. The experimental design involved persona-guided prompting to assess the VLM's ability to generate critiques from diverse perspectives. Our findings reveal the current performance levels, strengths, and areas for improvement of VLMs in the domain of art critique, offering insights into their potential and limitations in complex semantic understanding and content generation tasks. The code used for our experiments can be publicly accessed at: https://github.com/yha9806/VULCA-EMNLP2025.

Paper Structure

This paper contains 55 sections, 6 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Missing Evaluation Dimensions for Cultural Reasoning.
  4. Limitations of Persona Conditioning Without Grounding.
  5. Gap in Multimodal Input–Interpretation Evaluation.
  6. Lack of Comparative Cultural Interventions Across Models.
  7. Methodology
  8. MHEB Construction and Annotation Process
  9. Feature Engineering from Human Expert Critiques
  10. Evaluation of Dimensions and Label System
  11. Construction and Definition of Critic Personas
  12. Task Definition
  13. Vector Space Representation and Visualization
  14. Multi-Model Comparative Evaluation
  15. Quantitative Modeling and Formalisms
  16. ...and 40 more sections

Figures (4)

  • Figure 1: Overview of the VULCA framework, illustrating its components and their interactions for structured evaluation and intervention in art criticism.
  • Figure 2: T-SNE visual representation of human expert art commentaries.
  • Figure 3: Impact of Persona and Knowledge Base Interventions on VLM Critiques: A comprehensive analysis comparing intervened VLM outputs with a human expert benchmark. Left: t-SNE and KDE plots visualize the semantic distribution of critiques from different sources (human experts, baseline VLMs, intervened VLMs). Right: A radar chart compares average capability scores across dimensions like Profound Insight and Logical Clarity.
  • Figure 4: Profiling Summary: A comparative visualization of Human Experts vs. VLMs across key textual features (left), mean profile alignment scores (center), and t-SNE projection of profile vectors (right).