Where Do Images Come From? Analyzing Captions to Geographically Profile Datasets

TL;DR

The paper presents a large-scale, caption-based geolocation framework to profile the geographic origins of image–caption pairs in three widely used vision–language datasets. By extracting location mentions from captions and mapping them to countries via an extract--retrieve--predict LLM pipeline (augmented with GeoNames) and an entity-presence filter, the authors quantify country-level representation across 20 entities and multilingual captions, revealing strong US/UK/Canada bias and underrepresentation of Africa and South America (correlated with nominal GDP, $\rho \approx 0.82$–$0.84$). They compare geo-profiles to real-world distributions, show varying regional diversity, and examine country-wise image-generation behavior with Stable Diffusion, finding realistic yet narrowly distributed outputs and regional stereotypes. Limitations include language coverage, translation effects, and dependence on caption signals, but the work provides actionable insights for improving geo-diversity in multimodal data curation and model training. Overall, the study demonstrates substantial geographic biases in training data and model generations, motivating more globally representative data collection and robust multilingual analysis.

Abstract

Recent studies show that text-to-image models often fail to generate geographically representative images, raising concerns about the representativeness of their training data and motivating the question: which parts of the world do these training examples come from? We geographically profile large-scale multimodal datasets by mapping image-caption pairs to countries based on location information extracted from captions using LLMs. Studying English captions from three widely used datasets (Re-LAION, DataComp1B, and Conceptual Captions) across $20$ common entities (e.g., house, flag), we find that the United States, the United Kingdom, and Canada account for $48.0\%$ of samples, while South American and African countries are severely under-represented with only $1.8\%$ and $3.8\%$ of images, respectively. We observe a strong correlation between a country's GDP and its representation in the data ($ρ= 0.82$). Examining non-English subsets for $4$ languages from the Re-LAION dataset, we find that representation skews heavily toward countries where these languages are predominantly spoken. Additionally, we find that higher representation does not necessarily translate to greater visual or semantic diversity. Finally, analyzing country-specific images generated by Stable Diffusion v1.3 trained on Re-LAION, we show that while generations appear realistic, they are severely limited in their coverage compared to real-world images.

Figures (16)

• Figure 1: Distribution of countries for house and flag in Re-LAION2B-en. We show the uneven distribution of countries worldwide for house- and flag-related image-caption pairs in the Re-LAION2B-en dataset, along with the top $10$ most frequent countries for each entity.
• Figure 2: The proposed approach. Given an image-caption pair of an entity, we first map captions to countries (using Gemini), then filter out images that do not contain the entity (using the entity presence classifier), and finally return the countries for resulting image-caption pairs.
• Figure 3: Distribution of the countries across different datasets, averaged across entities. All datasets with English captions overrepresent countries like the US, UK, Canada, and under represent Afrian and South American nations. For the Japanese captions, Japan is the most represented country.
• Figure 4: Country distributions exhibit long-tailed behavior across the different datasets. This trend is visible for individual entities like road, house and flag, as well as for all entities combined.
• Figure 5: Distribution of the five most frequent countries in Re-LAION2B-en along with DataComp1B and CC12M, averaged across entities.