Fine-Tuned LLMs are "Time Capsules" for Tracking Societal Bias Through Books

TL;DR

This work treats large language models as time capsules that reflect evolving societal biases encoded in literature. By assembling BookPAGE, a decade-stratified corpus of 593 bestselling fictional books from 1950–2019, and fine-tuning multiple LLMs on decade-specific data, the authors probe bias across gender, sexual orientation, race, and religion via targeted prompts. The results show systematic bias shifts that align with historical contexts, with biases primarily arising from training content rather than model architecture. The approach offers a scalable framework for tracing temporal bias in AI and underscores the importance of diverse, temporally representative training data to promote fairer AI systems.

Abstract

Books, while often rich in cultural insights, can also mirror societal biases of their eras - biases that Large Language Models (LLMs) may learn and perpetuate during training. We introduce a novel method to trace and quantify these biases using fine-tuned LLMs. We develop BookPAGE, a corpus comprising 593 fictional books across seven decades (1950-2019), to track bias evolution. By fine-tuning LLMs on books from each decade and using targeted prompts, we examine shifts in biases related to gender, sexual orientation, race, and religion. Our findings indicate that LLMs trained on decade-specific books manifest biases reflective of their times, with both gradual trends and notable shifts. For example, model responses showed a progressive increase in the portrayal of women in leadership roles (from 8% to 22%) from the 1950s to 2010s, with a significant uptick in the 1990s (from 4% to 12%), possibly aligning with third-wave feminism. Same-sex relationship references increased markedly from the 1980s to 2000s (from 0% to 10%), mirroring growing LGBTQ+ visibility. Concerningly, negative portrayals of Islam rose sharply in the 2000s (26% to 38%), likely reflecting post-9/11 sentiments. Importantly, we demonstrate that these biases stem mainly from the books' content and not the models' architecture or initial training. Our study offers a new perspective on societal bias trends by bridging AI, literary studies, and social science research.

Figures (13)

• Figure 1: Temporal trends in Llama 3 70B's responses to religious associations with monument defacement. The lines represent second-degree polynomial best-fits.
• Figure 2: Overview of methodology from corpus creation to bias analysis, with section references
• Figure 4: Standard deviation of subcategory frequencies for each demographic across all decades
• Figure 5: Temporal trends in Llama 3 70B's responses to gender associations with CEOs. The lines represent second-degree polynomial best-fit curves.
• Figure 6: Temporal trends in Llama 3 70B's responses to sexual orientation associations with a woman's fiancé. The lines represent second-degree polynomial best-fit curves.