Detecting Linguistic Bias in Government Documents Using Large language Models

TL;DR

This study introduces the DGDB dataset, a bias-detection resource drawn from Dutch government documents, and demonstrates that fine-tuned Dutch BERT-based models achieve strong in-domain performance compared to generative LMs. By evaluating in-domain and out-of-domain generalization and applying resampling and explainability analyses, the work highlights both the potential and limitations of current models for detecting linguistic bias in governance texts. The authors provide a detailed annotation protocol, dataset statistics, and error analyses (via LIME) to elucidate term- and context-driven biases, underscoring the need for labeled, domain-specific data. The findings have practical implications for more equitable governance and offer a path toward expanding bias-detection research to other languages and contexts, with ethical considerations and SDG relevance discussed throughout.

Abstract

This paper addresses the critical need for detecting bias in government documents, an underexplored area with significant implications for governance. Existing methodologies often overlook the unique context and far-reaching impacts of governmental documents, potentially obscuring embedded biases that shape public policy and citizen-government interactions. To bridge this gap, we introduce the Dutch Government Data for Bias Detection (DGDB), a dataset sourced from the Dutch House of Representatives and annotated for bias by experts. We fine-tune several BERT-based models on this dataset and compare their performance with that of generative language models. Additionally, we conduct a comprehensive error analysis that includes explanations of the models' predictions. Our findings demonstrate that fine-tuned models achieve strong performance and significantly outperform generative language models, indicating the effectiveness of DGDB for bias detection. This work underscores the importance of labeled datasets for bias detection in various languages and contributes to more equitable governance practices.

Figures (8)

• Figure 1: Dataset Creation Process. The initial automated step (highlighted in blue) utilized bias-related keywords from established definition sources to query a web API and retrieve relevant documents. Subsequently, these normalized documents were iteratively sampled and annotated by bias detection experts in the second step (highlighted in red).
• Figure 2: Prompts for generative LMs in Dutch (left) and their English translation (right).
• Figure 3: Accuracy of bias detection over terms, relative to term occurrence (in-domain).
• Figure 4: Accuracy of bias detection over terms, relative to term occurence (out-of-domain).
• Figure 5: Confusion matrix for BERTje w/ undersampling.

Theorems & Definitions (1)

• definition 1