How far can bias go? -- Tracing bias from pretraining data to alignment

TL;DR

The correlation between gender-occupation bias in pre-training data and their manifestation in LLMs is examined, focusing on the Dolma dataset and the OLMo model, to reveal that biases present in pre-training data are amplified in model outputs.

Abstract

As LLMs are increasingly integrated into user-facing applications, addressing biases that perpetuate societal inequalities is crucial. While much work has gone into measuring or mitigating biases in these models, fewer studies have investigated their origins. Therefore, this study examines the correlation between gender-occupation bias in pre-training data and their manifestation in LLMs, focusing on the Dolma dataset and the OLMo model. Using zero-shot prompting and token co-occurrence analyses, we explore how biases in training data influence model outputs. Our findings reveal that biases present in pre-training data are amplified in model outputs. The study also examines the effects of prompt types, hyperparameters, and instruction-tuning on bias expression, finding instruction-tuning partially alleviating representational bias while still maintaining overall stereotypical gender associations, whereas hyperparameters and prompting variation have a lesser effect on bias expression. Our research traces bias throughout the LLM development pipeline and underscores the importance of mitigating bias at the pretraining stage.

Figures (8)

• Figure 1: Representation of women across 220 occupations, according to U.S. BLS (U.S. Bureau of Labor Statistics), in the Dolma dataset, and in outputs by OLMo 7B (base), OLMo 7B SFT and OLMo 7B Instruct, averaged across setups and prompts.
• Figure 2: Percentage of women- and men-oriented texts per occupational sector in the investigated Dolma sample according to sectors defined by the U.S. BLS.
• Figure 3: Percentage of women- and men-oriented texts generated by the OLMo models per occupational sector, averaged over all settings.
• Figure 4: Bias (De-)Amplification in the generated texts per model. The x-axis corresponds to the % women-occupation co-occurrences in the Dolma sample, and the y-axis corresponds to the % female-associated documents in the OLMo outputs. Each point represents an occupation. Shading: Amplification and (de-)amplification. Five occupational sectors are highlighted by color: Cleaning, Farming, fishing and forestry, Construction, and extraction, Iasnstallation and repair, Life and social sciences.
• Figure 5: Heat-maps depicting the Pearson correlation coefficient ($\rho$) between pre-training data and OLMo 7B base outputs averaged across decoding strategies and prompt types.