The Lou Dataset -- Exploring the Impact of Gender-Fair Language in German Text Classification

TL;DR

Lou introduces a high-quality reformulation dataset to assess the impact of gender-fair language on German text classification, covering seven tasks with 3.6k reformulations across three base datasets. The authors evaluate 16 LM variants under fine-tuning and in-context learning, revealing that gender-fair language can flip labels and lower certainty while largely preserving LM rankings across original and reformulated data. Key contributions include a rigorous annotation pipeline with amateur and professional validation, detailed layer- and attention-level analyses, and practical guidance for evaluating gender-fair language effects in NLP systems. The findings suggest gender-fair formulations induce syntactic shifts that propagate through LM processing, with implications for cross-language adoption and fair NLP development.

Abstract

Gender-fair language, an evolving German linguistic variation, fosters inclusion by addressing all genders or using neutral forms. Nevertheless, there is a significant lack of resources to assess the impact of this linguistic shift on classification using language models (LMs), which are probably not trained on such variations. To address this gap, we present Lou, the first dataset featuring high-quality reformulations for German text classification covering seven tasks, like stance detection and toxicity classification. Evaluating 16 mono- and multi-lingual LMs on Lou shows that gender-fair language substantially impacts predictions by flipping labels, reducing certainty, and altering attention patterns. However, existing evaluations remain valid, as LM rankings of original and reformulated instances do not significantly differ. While we offer initial insights on the effect on German text classification, the findings likely apply to other languages, as consistent patterns were observed in multi-lingual and English LMs.

Figures (17)

• Figure 1: A German stance detection instance from the Lou dataset. We reformulate the masculine formulation Konsumenten (consumers) regarding six inclusive or neutral strategies, highlighted in yellow. Translation: Consumers must be well supported.
• Figure 2: Mean performance and standard deviation, averaged over the seven Lou tasks and seeds (fine-tuning) or prompting templates (ICL) by model type (left) or specific LM (right).
• Figure 3: Difference between original and reformulated instances for strategies, model types, and tasks in average $F_1$ macro (left). The size and the color indicate the difference, whether positive (blue) or negative (red). On the right, we stack the average difference per LM and seed or prompt template for the model types and strategies.
• Figure 4: Label flip fractions for strategies, model types, and tasks. Size indicates the label flip fraction under gender-fair language and the color positive (blue) or negative (red) effect on aggregated performance.
• Figure 5: Performance difference and flip fraction against LMs' $F_1$ macros score of each task and strategy.