SoS: Analysis of Surface over Semantics in Multilingual Text-To-Image Generation

TL;DR

This work tackles the problem of language-induced surface biases in multilingual text-to-image (T2I) generation by formalizing Surface-over-Semantics (SoS) and proposing a language-agnostic SoS score built on image embeddings. It constructs a multilingual dataset spanning 171 cultures and 14 languages, prompts seven T2I models with fixed seeds, and analyzes outputs using the SoS metric, layer-wise encodings, and VQA-based visual descriptors. The key finding is that all but one model exhibit strong surface tendencies in at least two languages, with biases amplified in higher text-encoder layers and frequently correlating with culturally stereotypical depictions; some languages and cultures exhibit stronger semantic grounding, notably in the AD model. The study provides a robust, image-centric evaluation framework that complements CLIP-based methods and offers a basis for guiding fairer multilingual T2I training, while revealing ethical considerations around representation and cross-cultural perception that must be addressed in deployment.

Abstract

Text-to-image (T2I) models are increasingly employed by users worldwide. However, prior research has pointed to the high sensitivity of T2I towards particular input languages - when faced with languages other than English (i.e., different surface forms of the same prompt), T2I models often produce culturally stereotypical depictions, prioritizing the surface over the prompt's semantics. Yet a comprehensive analysis of this behavior, which we dub Surface-over-Semantics (SoS), is missing. We present the first analysis of T2I models' SoS tendencies. To this end, we create a set of prompts covering 171 cultural identities, translated into 14 languages, and use it to prompt seven T2I models. To quantify SoS tendencies across models, languages, and cultures, we introduce a novel measure and analyze how the tendencies we identify manifest visually. We show that all but one model exhibit strong surface-level tendency in at least two languages, with this effect intensifying across the layers of T2I text encoders. Moreover, these surface tendencies frequently correlate with stereotypical visual depictions.

Figures (59)

• Figure 1: Overview of our evaluation setup: (1) Constructing and translating each prompt into 13 other languages; (2) Generating corresponding images using one of 7 T2I models; (3) Analyzing output images through the SoS Score, a color analysis, and an analysis of commonly occurring descriptive terms.
• Figure 2: SoS Score Distribution with upper and lower quantiles, along with the mean (dotted line) and median (solid line) for each model.
• Figure 3: SoS Score Heatmap averaged across templates and person terms for AltDiffusion (left) and FLUX (right). Rows depict each culture, and columns are sorted by the mean SoS score per language.
• Figure 4: Averaged SoS scores per language across different text encoder layers. Colors indicate input languages, shown for (a) K3 and (b) SD21.
• Figure 5: Pearson correlation of SoS scores between languages across all models and cultural identities.