Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How Gender Interacts with Political Values: A Case Study on Czech BERT Models

Adnan Al Ali, Jindřich Libovický

Abstract

Neural language models, which reach state-of-the-art results on most natural language processing tasks, are trained on large text corpora that inevitably contain value-burdened content and often capture undesirable biases, which the models reflect. This case study focuses on the political biases of pre-trained encoders in Czech and compares them with a representative value survey. Because Czech is a gendered language, we also measure how the grammatical gender coincides with responses to men and women in the survey. We introduce a novel method for measuring the model's perceived political values. We find that the models do not assign statement probability following value-driven reasoning, and there is no systematic difference between feminine and masculine sentences. We conclude that BERT-sized models do not manifest systematic alignment with political values and that the biases observed in the models are rather due to superficial imitation of training data patterns than systematic value beliefs encoded in the models.

How Gender Interacts with Political Values: A Case Study on Czech BERT Models

Abstract

Neural language models, which reach state-of-the-art results on most natural language processing tasks, are trained on large text corpora that inevitably contain value-burdened content and often capture undesirable biases, which the models reflect. This case study focuses on the political biases of pre-trained encoders in Czech and compares them with a representative value survey. Because Czech is a gendered language, we also measure how the grammatical gender coincides with responses to men and women in the survey. We introduce a novel method for measuring the model's perceived political values. We find that the models do not assign statement probability following value-driven reasoning, and there is no systematic difference between feminine and masculine sentences. We conclude that BERT-sized models do not manifest systematic alignment with political values and that the biases observed in the models are rather due to superficial imitation of training data patterns than systematic value beliefs encoded in the models.
Paper Structure (17 sections, 4 equations, 2 figures, 3 tables)

This paper contains 17 sections, 4 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Prompting the Models
  5. Prompt Structure.
  6. Calibration Dataset.
  7. Agree vs. Disagree Correlation.
  8. Rescoring the Statements
  9. Comparison to Real-world Data
  10. Evaluated Models
  11. Results and Discussion
  12. Conclusions
  13. Acknowledgments
  14. Ethics Statement
  15. Bibliographical References
  16. ...and 2 more sections

Figures (2)

  • Figure 1: The correlation between the log-probability of the token(s) corresponding to "disagree" vs. "agree" in feminine sentences rated by the RobeCzech model. The identity function ($a=1$) shows that, in most cases, the model tends to rank "agree" with a higher probability.
  • Figure 2: The distribution of the models' ratings (• RobeCzech, $\blacksquare$ Czert, * FERNET News, $\blacktriangle$ mBERT, $\blacklozenge$ Slavic BERT, + XLM-R) of political statements in their feminine version grouped by the political values they correspond to: AntiAuth (bottom), CultLib, EconEq, Trib (top).