Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Relating Word Embedding Gender Biases to Gender Gaps: A Cross-Cultural Analysis

Scott Friedman, Sonja Schmer-Galunder, Anthony Chen, Jeffrey Rye

TL;DR

The paper introduces an axis-projection method to quantify gender bias in word embeddings and uses Twitter-derived, culture-specific embeddings to relate linguistic biases to broad gender-gap statistics across 99 countries and 51 U.S. states. By employing thematically grouped word sets and evaluating multiple embedding algorithms, the authors show that certain biased language dimensions align with corresponding gender gaps (internationally and within the U.S.), with stronger signals when using Word2Vec and the axis-projection metric. They further analyze adjectives using valence and dominance to reveal attitudinal patterns associated with reduced or increased gaps, and demonstrate the approach's potential to illuminate cultural attitudes from language data. Limitations include language constraints and correlational rather than causal findings, with future work targeting multilingual data and causal modeling to improve interpretation and applicability.

Abstract

Modern models for common NLP tasks often employ machine learning techniques and train on journalistic, social media, or other culturally-derived text. These have recently been scrutinized for racial and gender biases, rooting from inherent bias in their training text. These biases are often sub-optimal and recent work poses methods to rectify them; however, these biases may shed light on actual racial or gender gaps in the culture(s) that produced the training text, thereby helping us understand cultural context through big data. This paper presents an approach for quantifying gender bias in word embeddings, and then using them to characterize statistical gender gaps in education, politics, economics, and health. We validate these metrics on 2018 Twitter data spanning 51 U.S. regions and 99 countries. We correlate state and country word embedding biases with 18 international and 5 U.S.-based statistical gender gaps, characterizing regularities and predictive strength.

Relating Word Embedding Gender Biases to Gender Gaps: A Cross-Cultural Analysis

TL;DR

The paper introduces an axis-projection method to quantify gender bias in word embeddings and uses Twitter-derived, culture-specific embeddings to relate linguistic biases to broad gender-gap statistics across 99 countries and 51 U.S. states. By employing thematically grouped word sets and evaluating multiple embedding algorithms, the authors show that certain biased language dimensions align with corresponding gender gaps (internationally and within the U.S.), with stronger signals when using Word2Vec and the axis-projection metric. They further analyze adjectives using valence and dominance to reveal attitudinal patterns associated with reduced or increased gaps, and demonstrate the approach's potential to illuminate cultural attitudes from language data. Limitations include language constraints and correlational rather than causal findings, with future work targeting multilingual data and causal modeling to improve interpretation and applicability.

Abstract

Modern models for common NLP tasks often employ machine learning techniques and train on journalistic, social media, or other culturally-derived text. These have recently been scrutinized for racial and gender biases, rooting from inherent bias in their training text. These biases are often sub-optimal and recent work poses methods to rectify them; however, these biases may shed light on actual racial or gender gaps in the culture(s) that produced the training text, thereby helping us understand cultural context through big data. This paper presents an approach for quantifying gender bias in word embeddings, and then using them to characterize statistical gender gaps in education, politics, economics, and health. We validate these metrics on 2018 Twitter data spanning 51 U.S. regions and 99 countries. We correlate state and country word embedding biases with 18 international and 5 U.S.-based statistical gender gaps, characterizing regularities and predictive strength.
Paper Structure (10 sections, 1 equation, 7 figures)

This paper contains 10 sections, 1 equation, 7 figures.

Table of Contents

  1. Introduction
  2. Gender Gaps and Statistics
  3. Training Data
  4. Experiments
  5. International Analysis
  6. U.S. State Analysis
  7. Algorithm Comparison
  8. Valence and Dominance Analysis
  9. Conclusions
  10. Limitations and Future Work.

Figures (7)

  • Figure 1: Correlation of themed neutral word sets' gender bias (columns) against categories of gender gaps from worldbank.org (rows). Values are $R^2$ coefficient of determination, where negation is added to indicate inverse correlation. The rightmost four word sets (r-1 to r-4) were randomly sampled from the vocabulary for comparison.
  • Figure 2: Correlation of country's gender bias of government words (x-axis; female association increases in positive direction) against the World Economic Forum's political empowerment gender gap index (y-axis; gender gap decreases in positive direction).
  • Figure 3: Correlation of themed neutral word sets' gender bias (columns) against U.S. gender disparity statistics from CDC, US Census Bureau, and Represent Women's 2018 Gender Parity Report (rows). Values are $R^2$ coefficient of determination, where negation is added to indicate inverse correlation. The rightmost four word sets (rand 1-4) were randomly sampled from the vocabulary for comparison.
  • Figure 4: Correlation of states' gender bias of threat words' (x-axis; female association is positive direction) against the pay gap reported by U.S. Census Bureau in 2016 (y-axis; pay gap decreases in positive direction).
  • Figure 5: Comparison of three bias metrics and four word embedding algorithms correlating themed word sets' gender bias with U.S. gender gap statistics. Unlike in Fig. \ref{['fig:intl-plot']}, we perform feature selection using all countries.
  • ...and 2 more figures