Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Binary Gender: Evaluating Gender-Inclusive Machine Translation with Ambiguous Attitude Words

Yijie Chen, Yijin Liu, Fandong Meng, Jinan Xu, Yufeng Chen, Jie Zhou

TL;DR

This work introduces AmbGIMT, a gender-inclusive MT benchmark that uses ambiguous attitude words to evaluate bias beyond binary genders and proposes the Emotional Attitude Score (EAS) to quantify attitude shifts during translation. It constructs a rich English-Chinese parallel dataset with 14 non-binary identity settings, incorporating authentic and synthesized sentences and slot-based identity replacements, and evaluates multiple backbones (including LLaMA-2-7B, Mistral-7B, MiniCPM-2B, and NLLB-200-3.3B). The findings show pronounced bias against non-binary identities in translation quality and attitude, while lexical constraints in prompts substantially reduce bias and improve translation, though room for improvement remains. The work also provides a public dataset, a pipeline for evaluating ambiguous attitude words, and analysis linking word-frequency patterns to stereotypical translations, offering a foundation for future bias mitigation in gender-inclusive MT.

Abstract

Gender bias has been a focal point in the study of bias in machine translation and language models. Existing machine translation gender bias evaluations are primarily focused on male and female genders, limiting the scope of the evaluation. To assess gender bias accurately, these studies often rely on calculating the accuracy of gender pronouns or the masculine and feminine attributes of grammatical gender via the stereotypes triggered by occupations or sentiment words ({\em i.e.}, clear positive or negative attitude), which cannot extend to non-binary groups. This study presents a benchmark AmbGIMT (Gender-Inclusive Machine Translation with Ambiguous attitude words), which assesses gender bias beyond binary gender. Meanwhile, we propose a novel process to evaluate gender bias based on the Emotional Attitude Score (EAS), which is used to quantify ambiguous attitude words. In evaluating three recent and effective open-source LLMs and one powerful multilingual translation-specific model, our main observations are: (1) The translation performance within non-binary gender contexts is markedly inferior in terms of translation quality and exhibits more negative attitudes than binary-gender contexts. (2) The analysis experiments indicate that incorporating constraint context in prompts for gender identity terms can substantially reduce translation bias, while the bias remains evident despite the presence of the constraints. The code is publicly available at \url{https://github.com/pppa2019/ambGIMT}.

Beyond Binary Gender: Evaluating Gender-Inclusive Machine Translation with Ambiguous Attitude Words

TL;DR

This work introduces AmbGIMT, a gender-inclusive MT benchmark that uses ambiguous attitude words to evaluate bias beyond binary genders and proposes the Emotional Attitude Score (EAS) to quantify attitude shifts during translation. It constructs a rich English-Chinese parallel dataset with 14 non-binary identity settings, incorporating authentic and synthesized sentences and slot-based identity replacements, and evaluates multiple backbones (including LLaMA-2-7B, Mistral-7B, MiniCPM-2B, and NLLB-200-3.3B). The findings show pronounced bias against non-binary identities in translation quality and attitude, while lexical constraints in prompts substantially reduce bias and improve translation, though room for improvement remains. The work also provides a public dataset, a pipeline for evaluating ambiguous attitude words, and analysis linking word-frequency patterns to stereotypical translations, offering a foundation for future bias mitigation in gender-inclusive MT.

Abstract

Gender bias has been a focal point in the study of bias in machine translation and language models. Existing machine translation gender bias evaluations are primarily focused on male and female genders, limiting the scope of the evaluation. To assess gender bias accurately, these studies often rely on calculating the accuracy of gender pronouns or the masculine and feminine attributes of grammatical gender via the stereotypes triggered by occupations or sentiment words ({\em i.e.}, clear positive or negative attitude), which cannot extend to non-binary groups. This study presents a benchmark AmbGIMT (Gender-Inclusive Machine Translation with Ambiguous attitude words), which assesses gender bias beyond binary gender. Meanwhile, we propose a novel process to evaluate gender bias based on the Emotional Attitude Score (EAS), which is used to quantify ambiguous attitude words. In evaluating three recent and effective open-source LLMs and one powerful multilingual translation-specific model, our main observations are: (1) The translation performance within non-binary gender contexts is markedly inferior in terms of translation quality and exhibits more negative attitudes than binary-gender contexts. (2) The analysis experiments indicate that incorporating constraint context in prompts for gender identity terms can substantially reduce translation bias, while the bias remains evident despite the presence of the constraints. The code is publicly available at \url{https://github.com/pppa2019/ambGIMT}.
Paper Structure (34 sections, 6 equations, 3 figures, 14 tables)

This paper contains 34 sections, 6 equations, 3 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Background and Pre-definition
  3. Background
  4. Emotional Attitude Evaluation
  5. AmbGIMT Dataset
  6. Gender-related Identities
  7. Seed Word Selection
  8. Dataset Construction
  9. Authentic Source Sentences
  10. Synthesized Source Sentences
  11. Translation Sentence and Replacing Datasets
  12. Data Statistic
  13. Experiemts
  14. Settings
  15. Main Results
  16. ...and 19 more sections

Figures (3)

  • Figure 1: Overall pipeline of data construction. We collect authentic and synthesized gender-neutral English sentences with ambiguous attitude words and then translate the sentences into Chinese. The slot terms of parallel gender-neutral sentence pairs are replaced by the rule-based method into identity settings across sex, gender, and sexual orientation (14 settings in total), and the final test set is obtained.
  • Figure 2: The EAS of the source ambiguous words and their translations. In the middle areas with the origin word attitude score from -2.5 to 0.8, the data points show higher variances and higher possibilities to change the emotional attitude type from positive to negative or the opposite situation.
  • Figure 3: Confusion matrix heatmap for the prediction result of MiniCPM-2B, Germma-7B-it and Mistral-7b.