How Culturally Aware are Vision-Language Models?

TL;DR

This paper addresses how culturally aware modern vision-language models are for image captioning of culture-rich imagery. It introduces a Cultural Awareness Score (CAS) and MOSAIC-1.5k dataset to quantify and benchmark culture-related content in captions. It evaluates four prominent VLMs (GPT-4V, Gemini Pro Vision, LLaVA, OpenFlamingo) and reveals generally low cultural awareness with notable model and category differences; real-world dances are better captured than signs/symbols, which also suffer from hallucinations. The work provides a publicly available dataset and CAS framework to guide future improvements toward culturally sensitive AI systems.

Abstract

An image is often considered worth a thousand words, and certain images can tell rich and insightful stories. Can these stories be told via image captioning? Images from folklore genres, such as mythology, folk dance, cultural signs, and symbols, are vital to every culture. Our research compares the performance of four popular vision-language models (GPT-4V, Gemini Pro Vision, LLaVA, and OpenFlamingo) in identifying culturally specific information in such images and creating accurate and culturally sensitive image captions. We also propose a new evaluation metric, the Cultural Awareness Score (CAS), which measures the degree of cultural awareness in image captions. We provide a dataset MOSAIC-1.5k labeled with ground truth for images containing cultural background and context and a labeled dataset with assigned Cultural Awareness Scores that can be used with unseen data. Creating culturally appropriate image captions is valuable for scientific research and can be beneficial for many practical applications. We envision our work will promote a deeper integration of cultural sensitivity in AI applications worldwide. By making the dataset and Cultural Awareness Score available to the public, we aim to facilitate further research in this area, encouraging the development of more culturally aware AI systems that respect and celebrate global diversity.

Figures (7)

• Figure 1: ROUGE-L across the models chosen for experimentation.
• Figure 2: Omamori is generally translated as an amulet, or good luck charm, and comes from the Japanese word mamoru.
• Figure 3: Ouroboros is an ancient symbol depicting a serpent or dragon eating its own tail, representing the eternal cycle, renewal, the unity of opposites, and eternity.
• Figure 4: Kathakali, one of the eight classical dance styles of India.
• Figure 5: Cornucopia represents abundance, wealth, and prosperity.