Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WMT24 Test Suite: Gender Resolution in Speaker-Listener Dialogue Roles

Hillary Dawkins, Isar Nejadgholi, Chi-kiu Lo

Abstract

We assess the difficulty of gender resolution in literary-style dialogue settings and the influence of gender stereotypes. Instances of the test suite contain spoken dialogue interleaved with external meta-context about the characters and the manner of speaking. We find that character and manner stereotypes outside of the dialogue significantly impact the gender agreement of referents within the dialogue.

WMT24 Test Suite: Gender Resolution in Speaker-Listener Dialogue Roles

Abstract

We assess the difficulty of gender resolution in literary-style dialogue settings and the influence of gender stereotypes. Instances of the test suite contain spoken dialogue interleaved with external meta-context about the characters and the manner of speaking. We find that character and manner stereotypes outside of the dialogue significantly impact the gender agreement of referents within the dialogue.

Paper Structure

This paper contains 13 sections, 9 equations, 3 figures, 29 tables.

Table of Contents

  1. Introduction
  2. Test Suite Description
  3. Methodology
  4. Gender-Stereotyped Manner
  5. Gender-Stereotyped Characters
  6. "Opposite" or "Same" Binary Gender Speaker Bias
  7. Future Work
  8. Conclusion
  9. Test Suite Templates
  10. Stereo-Adverb Templates
  11. Stereo-Character Templates
  12. Structure Templates
  13. Results for All Systems

Figures (3)

  • Figure 1: Gender-stereotyped adverbs outside of the dialogue affect the adjective's gender agreement with the speaker within the dialogue. Source sentences in English include instances without adverbs (a) and with stereotypically masculine (b) or feminine adverbs (c). When translated to the target language, adjectives tend to align with the stereotype (adjectives shown here in Czech).
  • Figure 2: Gender-stereotyped character descriptions outside of the dialogue affect the adjective's gender agreement.
  • Figure 3: The opposite binary gender effect is present in both ambiguous (a) and determined (b) cases. Determined cases with a single known gender (c) are unchallenging despite having the same structural components (i.e. both speaker (I) and listener (you) resolutions, and need to "look ahead" in the text to find the adjective's referent). All effects are the same but flipped for systems that prefer same-gender speaker pairs.