Towards Automatic Evaluation for Image Transcreation

TL;DR

This paper tackles the lack of automatic evaluation for image transcreation by introducing three metric families—object-based CSI-Overlap, embedding-based, and VLM-based—each capturing cultural relevance, semantic equivalence, and visual similarity. Grounded in translation studies, it defines the three axes and demonstrates through cross-country meta-evaluation that a hybrid approach leveraging both VLMs and embedding models provides robust evaluation signals, with correlations to human judgments ranging up to 0.87 for visual similarity. The study also discusses practical strengths, limitations, and recommendations, offering a framework and codebase to accelerate research in automated image transcreation. Overall, it advances scalable, theory-informed evaluation for culturally adaptive visual localization with potential impact on localization pipelines and dataset benchmarking.

Abstract

Beyond conventional paradigms of translating speech and text, recently, there has been interest in automated transcreation of images to facilitate localization of visual content across different cultures. Attempts to define this as a formal Machine Learning (ML) problem have been impeded by the lack of automatic evaluation mechanisms, with previous work relying solely on human evaluation. In this paper, we seek to close this gap by proposing a suite of automatic evaluation metrics inspired by machine translation (MT) metrics, categorized into: a) Object-based, b) Embedding-based, and c) VLM-based. Drawing on theories from translation studies and real-world transcreation practices, we identify three critical dimensions of image transcreation: cultural relevance, semantic equivalence and visual similarity, and design our metrics to evaluate systems along these axes. Our results show that proprietary VLMs best identify cultural relevance and semantic equivalence, while vision-encoder representations are adept at measuring visual similarity. Meta-evaluation across 7 countries shows our metrics agree strongly with human ratings, with average segment-level correlations ranging from 0.55-0.87. Finally, through a discussion of the merits and demerits of each metric, we offer a robust framework for automated image transcreation evaluation, grounded in both theoretical foundations and practical application. Our code can be found here: https://github.com/simran-khanuja/automatic-eval-img-transcreation.

Figures (6)

• Figure 1: Image Transcreation in the real world: The Simpsons (1989) was remade into Al-Shamshoon (2005) for a Middle-Eastern audience.
• Figure 2: The Coca-Cola homepage depicts different cultural concepts in different countries cocacolaad. The top row shows Marvel-themed ads where visuals are consistent but the text is translated for accessibility, while the bottom one depicts the Lunar New Year to celebrate the Year of the Rabbit in Korea.
• Figure 3: Both Slack ads convey how the product can be used to facilitate hybrid work, but the names and avatars of users are localized to reflect the demographics of each country. Figures sourced from slackblog.
• Figure 4: Examples of transcreation to adhere to cultural norms. Left: Tarzan covers his thighs with culturally appropriate clothing; Right: woman is completely dressed in black.
• Figure 5: Transcreation reflecting cultural differences -- in the U.S. adaptation of Sailor Moon, Artemis is portrayed as cunning and mischievous ("catty"), while in Japan, cats are seen as part of the cute kawaii culture.