Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gender Inflected or Bias Inflicted: On Using Grammatical Gender Cues for Bias Evaluation in Machine Translation

Pushpdeep Singh

TL;DR

This work uses Hindi as the source language and constructs two sets of gender-specific sentences that are used to evaluate different Hindi-English (HI-EN) NMT systems automatically for gender bias, highlighting the importance of considering the nature of language when designing such extrinsic bias evaluation datasets.

Abstract

Neural Machine Translation (NMT) models are state-of-the-art for machine translation. However, these models are known to have various social biases, especially gender bias. Most of the work on evaluating gender bias in NMT has focused primarily on English as the source language. For source languages different from English, most of the studies use gender-neutral sentences to evaluate gender bias. However, practically, many sentences that we encounter do have gender information. Therefore, it makes more sense to evaluate for bias using such sentences. This allows us to determine if NMT models can identify the correct gender based on the grammatical gender cues in the source sentence rather than relying on biased correlations with, say, occupation terms. To demonstrate our point, in this work, we use Hindi as the source language and construct two sets of gender-specific sentences: OTSC-Hindi and WinoMT-Hindi that we use to evaluate different Hindi-English (HI-EN) NMT systems automatically for gender bias. Our work highlights the importance of considering the nature of language when designing such extrinsic bias evaluation datasets.

Gender Inflected or Bias Inflicted: On Using Grammatical Gender Cues for Bias Evaluation in Machine Translation

TL;DR

This work uses Hindi as the source language and constructs two sets of gender-specific sentences that are used to evaluate different Hindi-English (HI-EN) NMT systems automatically for gender bias, highlighting the importance of considering the nature of language when designing such extrinsic bias evaluation datasets.

Abstract

Neural Machine Translation (NMT) models are state-of-the-art for machine translation. However, these models are known to have various social biases, especially gender bias. Most of the work on evaluating gender bias in NMT has focused primarily on English as the source language. For source languages different from English, most of the studies use gender-neutral sentences to evaluate gender bias. However, practically, many sentences that we encounter do have gender information. Therefore, it makes more sense to evaluate for bias using such sentences. This allows us to determine if NMT models can identify the correct gender based on the grammatical gender cues in the source sentence rather than relying on biased correlations with, say, occupation terms. To demonstrate our point, in this work, we use Hindi as the source language and construct two sets of gender-specific sentences: OTSC-Hindi and WinoMT-Hindi that we use to evaluate different Hindi-English (HI-EN) NMT systems automatically for gender bias. Our work highlights the importance of considering the nature of language when designing such extrinsic bias evaluation datasets.
Paper Structure (12 sections, 1 equation, 3 figures, 3 tables)

This paper contains 12 sections, 1 equation, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Experimental Setup
  3. TGBI Evaluation using Gender-Neutral Sentences
  4. Approach
  5. OTSC-Hindi
  6. WinoMT-Hindi
  7. Results and Discussion
  8. TGBI Evaluation
  9. Evaluation using OTSC-Hindi
  10. Evaluation using WinoMT-Hindi
  11. Related Work
  12. Conclusion and Future Work

Figures (3)

  • Figure 1: OTSC-Hindi sample template sentence along with its English translation. Gender of the speaker is specified by gender-inflected verb, i.e. " jAntA" or " jAntF". The possessive pronoun " m?rA" or " m?rF" and the verb " krtA" or " krtF" specify friend's gender. Here, the pronoun " us?" references speaker's friend.
  • Figure 2: Sentence Template for WinoMT-Hindi. When Entity 1 is referenced, we use gender-inflected verb to specify its gender. When Entity 2 is referenced, its gender is specified using gender-inflected relational postposition or an adjective. Phrase after the conjuction (containing the pronoun which refers to either entity) is gender neutral.
  • Figure 3: Sample Sentences in WinoMT-Hindi. The solid line shows pro-stereotypical coreference, while the dashed line shows anti-stereotypical coreference. Male and female (stereotypically) entities are marked in blue and orange boxes, respectively. Hindi pronouns are marked in blue or orange box based on the actual gender of their referred entity according to the grammatical context.