Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multiple-Debias: A Full-process Debiasing Method for Multilingual Pre-trained Language Models

Haoyu Liang, Peijian Zeng, Wentao Huang, Aimin Yang, Dong Zhou

Abstract

Multilingual Pre-trained Language Models (MPLMs) have become essential tools for natural language processing. However, they often exhibit biases related to sensitive attributes such as gender, race, and religion. In this paper, we introduce a comprehensive multilingual debiasing method named Multiple-Debias to address these issues across multiple languages. By incorporating multilingual counterfactual data augmentation and multilingual Self-Debias across both pre-processing and post-processing stages, alongside parameter-efficient fine-tuning, we significantly reduced biases in MPLMs across three sensitive attributes in four languages. We also extended CrowS-Pairs to German, Spanish, Chinese, and Japanese, validating our full-process multilingual debiasing method for gender, racial, and religious bias. Our experiments show that (i) multilingual debiasing methods surpass monolingual approaches in effectively mitigating biases, and (ii) integrating debiasing information from different languages notably improves the fairness of MPLMs.

Multiple-Debias: A Full-process Debiasing Method for Multilingual Pre-trained Language Models

Abstract

Multilingual Pre-trained Language Models (MPLMs) have become essential tools for natural language processing. However, they often exhibit biases related to sensitive attributes such as gender, race, and religion. In this paper, we introduce a comprehensive multilingual debiasing method named Multiple-Debias to address these issues across multiple languages. By incorporating multilingual counterfactual data augmentation and multilingual Self-Debias across both pre-processing and post-processing stages, alongside parameter-efficient fine-tuning, we significantly reduced biases in MPLMs across three sensitive attributes in four languages. We also extended CrowS-Pairs to German, Spanish, Chinese, and Japanese, validating our full-process multilingual debiasing method for gender, racial, and religious bias. Our experiments show that (i) multilingual debiasing methods surpass monolingual approaches in effectively mitigating biases, and (ii) integrating debiasing information from different languages notably improves the fairness of MPLMs.

Paper Structure

This paper contains 13 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Multiple-Debias
  3. Multilingual Counterfactual Data Augmentation
  4. Parameter-Efficient Fine-Tuning
  5. Multilingual-Self-Debias
  6. Multilingual Fairness Evaluation
  7. Multilingual CrowS-Pairs
  8. Multilingual Bias Evaluation
  9. Experiments and Results
  10. Experimental Setup
  11. Results of Mitigating Gender Bias
  12. Results of Mitigating Racial and Religious Bias
  13. Conclusion

Figures (3)

  • Figure 1: The framework of Multiple-Debias.
  • Figure 2: MBE bias scores of various debiasing methods on the mBERT model.
  • Figure 3: MBE bias scores of various debiasing methods on the XLM-R model.