Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Pre-Trained Language Models Represent Some Geographic Populations Better Than Others

Jonathan Dunn, Benjamin Adams, Harish Tayyar Madabushi

TL;DR

This study quantify geographic population skew in two families of pre-trained language models (OPT and BLOOM) using a geo-referenced, English-language probing task across 927 locations and 130 countries. It evaluates perplexity-based representativeness of locality sub-corpora for multiple model sizes, revealing a robust skew favoring North American and UK populations with poorer representation in South and Southeast Asia. The skew is highly concordant across model families and sizes, and cannot be fully explained by simple sociolinguistic, economic, or geographic factors, implying training data geography remains a hidden factor. The findings challenge the validity of a single universal model for all populations and motivate explicit population adaptation and transparent mapping of training data geography for equitable language technologies; future work points to population-aware mapping efforts such as earthLings.io.

Abstract

This paper measures the skew in how well two families of LLMs represent diverse geographic populations. A spatial probing task is used with geo-referenced corpora to measure the degree to which pre-trained language models from the OPT and BLOOM series represent diverse populations around the world. Results show that these models perform much better for some populations than others. In particular, populations across the US and the UK are represented quite well while those in South and Southeast Asia are poorly represented. Analysis shows that both families of models largely share the same skew across populations. At the same time, this skew cannot be fully explained by sociolinguistic factors, economic factors, or geographic factors. The basic conclusion from this analysis is that pre-trained models do not equally represent the world's population: there is a strong skew towards specific geographic populations. This finding challenges the idea that a single model can be used for all populations.

Pre-Trained Language Models Represent Some Geographic Populations Better Than Others

TL;DR

This study quantify geographic population skew in two families of pre-trained language models (OPT and BLOOM) using a geo-referenced, English-language probing task across 927 locations and 130 countries. It evaluates perplexity-based representativeness of locality sub-corpora for multiple model sizes, revealing a robust skew favoring North American and UK populations with poorer representation in South and Southeast Asia. The skew is highly concordant across model families and sizes, and cannot be fully explained by simple sociolinguistic, economic, or geographic factors, implying training data geography remains a hidden factor. The findings challenge the validity of a single universal model for all populations and motivate explicit population adaptation and transparent mapping of training data geography for equitable language technologies; future work points to population-aware mapping efforts such as earthLings.io.

Abstract

This paper measures the skew in how well two families of LLMs represent diverse geographic populations. A spatial probing task is used with geo-referenced corpora to measure the degree to which pre-trained language models from the OPT and BLOOM series represent diverse populations around the world. Results show that these models perform much better for some populations than others. In particular, populations across the US and the UK are represented quite well while those in South and Southeast Asia are poorly represented. Analysis shows that both families of models largely share the same skew across populations. At the same time, this skew cannot be fully explained by sociolinguistic factors, economic factors, or geographic factors. The basic conclusion from this analysis is that pre-trained models do not equally represent the world's population: there is a strong skew towards specific geographic populations. This finding challenges the idea that a single model can be used for all populations.
Paper Structure (8 sections, 8 figures, 5 tables)

This paper contains 8 sections, 8 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Language Data
  4. Methods
  5. Results and Analysis
  6. Discussion and Conclusions
  7. Ethics Statement
  8. Bibliographical References

Figures (8)

  • Figure 1: Map of Local Populations. Each point represents a data collection location.
  • Figure 2: Countries by standardized perplexity scores across sub-corpora for BigScience bloom 3b. Because the z-score is used, 1 reflects one standard deviation above the mean and -1 below the mean.
  • Figure 3: Local populations grouped by region by standardized perplexity scores for Facebook opt 2.7b
  • Figure 4: Top-100 lowest (blue) and highest (red) mean perplexity locations, Facebook opt 2.7b.
  • Figure 6: Countries by standard deviation in standardized perplexity scores across sub-corpora for BigScience bloom 3b. Higher standard deviations indicate more variation within a country.
  • ...and 3 more figures