MisgenderMender: A Community-Informed Approach to Interventions for Misgendering

TL;DR

MisgenderMender tackles the gap in automated interventions for misgendering by first engaging gender-diverse individuals in the US to understand their preferences and concerns. The authors define a two-subtask framework—detecting misgendering and, where appropriate, editing it—and build the MisgenderMender dataset, comprising 3790 annotated instances from social media posts and LLM-generated content about 30 public figures with publicly available gender profiles. They establish baseline detection and editing performance across domains, with GPT-4 achieving the strongest detection and a high, but improvable, edit accuracy (97% corrections with 4.6% unnecessary edits). The work emphasizes community-informed design, domain-sensitive interventions, and ethical safeguards, and releases the dataset and code to spur further research toward responsible, transparent misgendering interventions.

Abstract

Content Warning: This paper contains examples of misgendering and erasure that could be offensive and potentially triggering. Misgendering, the act of incorrectly addressing someone's gender, inflicts serious harm and is pervasive in everyday technologies, yet there is a notable lack of research to combat it. We are the first to address this lack of research into interventions for misgendering by conducting a survey of gender-diverse individuals in the US to understand perspectives about automated interventions for text-based misgendering. Based on survey insights on the prevalence of misgendering, desired solutions, and associated concerns, we introduce a misgendering interventions task and evaluation dataset, MisgenderMender. We define the task with two sub-tasks: (i) detecting misgendering, followed by (ii) correcting misgendering where misgendering is present in domains where editing is appropriate. MisgenderMender comprises 3790 instances of social media content and LLM-generations about non-cisgender public figures, annotated for the presence of misgendering, with additional annotations for correcting misgendering in LLM-generated text. Using this dataset, we set initial benchmarks by evaluating existing NLP systems and highlighting challenges for future models to address. We release the full dataset, code, and demo at https://tamannahossainkay.github.io/misgendermender/.

Figures (5)

• Figure 1: MisgenderMender examples consisting of a gender linguistic profile and corresponding annotated content for detecting and correcting misgendering.
• Figure 2: Survey responses Count of participants (out of 33) reporting experiences with misgendering and expressing a desire for detection, correction, or hiding of misgendering across various domains.
• Figure 3: Problem Setup The misgendering interventions task can be divided into two sub-tasks: (i) detecting misgendering, followed by (ii) correcting misgendering, in domains where editing is appropriate.
• Figure 4: MTurk Instructions Instructions provided to MTurk annotators to annotate LLM-generated content. Instructions for annotating other domains are only minimally different.
• Figure 5: MTurk Interface Here we present the interface for annotating a single instance of LLM-generated content.