Evaluating Short-Term Temporal Fluctuations of Social Biases in Social Media Data and Masked Language Models

TL;DR

This work investigates whether social biases encoded in masked language models trained on temporally evolving social-media data drift over time. By leveraging temporally sliced corpora from X and a series of RoBERTa-based TimeLMs, evaluated with the intrinsic AULA bias metric on CrowS-Pairs and StereoSet, the authors quantify both overall and type-specific biases across 2020–2022, and compare them to biases present in the training data. They find that while overall bias remains relatively stable, several bias types (e.g., race color, religion, sexual orientation) fluctuate over time, and biases in data often reflect these nuances; correlations among bias types further reveal nonuniform temporal dynamics. The results highlight the importance of per-bias-type evaluation rather than relying solely on aggregate bias scores for safe deployment and call for broader analyses across architectures, languages, and data sources in future work.

Abstract

Social biases such as gender or racial biases have been reported in language models (LMs), including Masked Language Models (MLMs). Given that MLMs are continuously trained with increasing amounts of additional data collected over time, an important yet unanswered question is how the social biases encoded with MLMs vary over time. In particular, the number of social media users continues to grow at an exponential rate, and it is a valid concern for the MLMs trained specifically on social media data whether their social biases (if any) would also amplify over time. To empirically analyse this problem, we use a series of MLMs pretrained on chronologically ordered temporal snapshots of corpora. Our analysis reveals that, although social biases are present in all MLMs, most types of social bias remain relatively stable over time (with a few exceptions). To further understand the mechanisms that influence social biases in MLMs, we analyse the temporal corpora used to train the MLMs. Our findings show that some demographic groups, such as male, obtain higher preference over the other, such as female on the training corpora constantly.

Figures (4)

• Figure 1: Social bias scores across time for different types of biases computed using the AULA metric. Results evaluated on the CrowS-Pairs and StereoSet datasets are shown respectively on the left and right. The 'bias score' (in dark blue) indicates the overall bias score.
• Figure 2: Pearson correlation coefficient of each pair of bias types. Results on the CrowS-Pairs and StereoSet datasets are shown respectively on the left and right.
• Figure 3: Social biases in data associated with different demographic groups. A sentiment classifier is used to determine whether a tweet associated with a particular demographic group conveys positive or negative sentiment. Dash line represents the bias scores computed using \ref{['eq:AULA']} on CrowS-Pairs, while solid lines show bias scores computed using \ref{['eq:bias-data']}, respectively.
• Figure 4: Social bias scores across time for different types of biases computed using the AULA metric for COHABERT models. Results evaluated on the CrowS-Pairs and StereoSet datasets are shown respectively on the top and bottom. The 'bias score' (in dark blue) indicates the overall bias score.