Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CultiVerse: Towards Cross-Cultural Understanding for Paintings with Large Language Model

Wei Zhang, Wong Kam-Kwai, Biying Xu, Yiwen Ren, Yuhuai Li, Minfeng Zhu, Yingchaojie Feng, Wei Chen

TL;DR

This paper tackles the challenge of cross-cultural understanding in art appreciation by integrating Large Language Models with a visual analytics interface. It introduces CultiVerse, a mixed-initiative system that combines an expert-curated TCP Cultural Norm Dataset with an interactive workflow to map and translate cultural symbols across cultures. The authors define a structured cultural-norm representation, develop a robust data pipeline from NPM and CIT, and validate the approach through a user-study with diverse participants, showing improved cross-cultural insight and engagement compared to baseline methods. The work demonstrates the potential of multimodal, prompt-engineered AI-assisted exploration for education, museum-guided interpretation, and intercultural communication, while outlining practical limitations and directions for broader adoption and future research.

Abstract

The integration of new technology with cultural studies enhances our understanding of cultural heritage but often struggles to connect with diverse audiences. It is challenging to align personal interpretations with the intended meanings across different cultures. Our study investigates the important factors in appreciating art from a cross-cultural perspective. We explore the application of Large Language Models (LLMs) to bridge the cultural and language barriers in understanding Traditional Chinese Paintings (TCPs). We present CultiVerse, a visual analytics system that utilizes LLMs within a mixed-initiative framework, enhancing interpretative appreciation of TCP in a cross-cultural dialogue. CultiVerse addresses the challenge of translating the nuanced symbolism in art, which involves interpreting complex cultural contexts, aligning cross-cultural symbols, and validating cultural acceptance. CultiVerse integrates an interactive interface with the analytical capability of LLMs to explore a curated TCP dataset, facilitating the analysis of multifaceted symbolic meanings and the exploration of cross-cultural serendipitous discoveries. Empirical evaluations affirm that CultiVerse significantly improves cross-cultural understanding, offering deeper insights and engaging art appreciation.

CultiVerse: Towards Cross-Cultural Understanding for Paintings with Large Language Model

TL;DR

This paper tackles the challenge of cross-cultural understanding in art appreciation by integrating Large Language Models with a visual analytics interface. It introduces CultiVerse, a mixed-initiative system that combines an expert-curated TCP Cultural Norm Dataset with an interactive workflow to map and translate cultural symbols across cultures. The authors define a structured cultural-norm representation, develop a robust data pipeline from NPM and CIT, and validate the approach through a user-study with diverse participants, showing improved cross-cultural insight and engagement compared to baseline methods. The work demonstrates the potential of multimodal, prompt-engineered AI-assisted exploration for education, museum-guided interpretation, and intercultural communication, while outlining practical limitations and directions for broader adoption and future research.

Abstract

The integration of new technology with cultural studies enhances our understanding of cultural heritage but often struggles to connect with diverse audiences. It is challenging to align personal interpretations with the intended meanings across different cultures. Our study investigates the important factors in appreciating art from a cross-cultural perspective. We explore the application of Large Language Models (LLMs) to bridge the cultural and language barriers in understanding Traditional Chinese Paintings (TCPs). We present CultiVerse, a visual analytics system that utilizes LLMs within a mixed-initiative framework, enhancing interpretative appreciation of TCP in a cross-cultural dialogue. CultiVerse addresses the challenge of translating the nuanced symbolism in art, which involves interpreting complex cultural contexts, aligning cross-cultural symbols, and validating cultural acceptance. CultiVerse integrates an interactive interface with the analytical capability of LLMs to explore a curated TCP dataset, facilitating the analysis of multifaceted symbolic meanings and the exploration of cross-cultural serendipitous discoveries. Empirical evaluations affirm that CultiVerse significantly improves cross-cultural understanding, offering deeper insights and engaging art appreciation.
Paper Structure (32 sections, 7 equations, 8 figures)

This paper contains 32 sections, 7 equations, 8 figures.

Table of Contents

  1. Introduction
  2. Related works
  3. Visual analytics for cultural understanding
  4. Facilitating cross-cultural understanding
  5. Incorporating with Large Language Models
  6. Design requirement analysis
  7. Painting analysis process
  8. Design requirement for cross-cultural understanding
  9. Source cultural norm dataset construction
  10. Cultural norm definition
  11. Data collection and annotation
  12. Norm extraction
  13. CultiVerse
  14. User background setting
  15. Source culture extraction component
  16. ...and 17 more sections

Figures (8)

  • Figure 1: The lotus flower carries diverse meanings in different cultural contexts. (A) The"Lotus Painting" by Xu Wei symbolizes nobility ruan2023lotus. (B) The "Lotus Pond and Mandarin Ducks" by Shen Quan expresses a harmonious love relationship sun2023lotus. (C) Painting from the Bhagavata Purana, a Hindu manuscript, describes creation hallman1954lotus.
  • Figure 2: The design requirements are inspired by TCP comprehension levels (A-C) and the literature translation process (D).
  • Figure 3: The CultiVerse workflow comprises three steps: source culture extraction (A), culture exchange (B), and target culture extrapolation (C). Initially, user observe element distribution (A1), examine element correlations (A2), and select painting based on exploratory interests (A3), followed by choosing element for analysis (A4). Subsequently, user analyze (B1), generate (B2), and translate (B3) cultural norms in conjunction with LLMs. Finally, users validate (C1) and expand (C2) the translation results.
  • Figure 4: The element selection view highlights the frequency of elements in the dataset (A) and counts of other factors in the cultural norm (B-F).
  • Figure 5: Case 2 explores (A1) "Autumn Egret in Hibiscus Flower Landscape." (A2) Analyzing compositional elements to (A3) understand their significance in Chinese culture; and (B1) using interactive annotations to (B2) explore new elements (B3) in Indonesian culture.
  • ...and 3 more figures