Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Social Biases in NLP Models as Barriers for Persons with Disabilities

Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, Stephen Denuyl

TL;DR

The paper investigates how NLP systems encode biases about persons with disabilities, showing biases in toxicity and sentiment classifiers, embeddings, and data co-occurrence with stigmatizing topics. It employs perturbation-based tests, BERT embedding probes, and dataset analysis to reveal where biases originate and how they manifest downstream. The findings highlight that disability terms can be unfairly associated with toxicity and negative sentiment, and that social discourse topics around homelessness, gun violence, and drug use amplify these biases. The authors argue for socio-technical mitigation, transparency, and inclusive, stakeholder-driven approaches to reduce harm in real-world deployments.

Abstract

Building equitable and inclusive NLP technologies demands consideration of whether and how social attitudes are represented in ML models. In particular, representations encoded in models often inadvertently perpetuate undesirable social biases from the data on which they are trained. In this paper, we present evidence of such undesirable biases towards mentions of disability in two different English language models: toxicity prediction and sentiment analysis. Next, we demonstrate that the neural embeddings that are the critical first step in most NLP pipelines similarly contain undesirable biases towards mentions of disability. We end by highlighting topical biases in the discourse about disability which may contribute to the observed model biases; for instance, gun violence, homelessness, and drug addiction are over-represented in texts discussing mental illness.

Social Biases in NLP Models as Barriers for Persons with Disabilities

TL;DR

The paper investigates how NLP systems encode biases about persons with disabilities, showing biases in toxicity and sentiment classifiers, embeddings, and data co-occurrence with stigmatizing topics. It employs perturbation-based tests, BERT embedding probes, and dataset analysis to reveal where biases originate and how they manifest downstream. The findings highlight that disability terms can be unfairly associated with toxicity and negative sentiment, and that social discourse topics around homelessness, gun violence, and drug use amplify these biases. The authors argue for socio-technical mitigation, transparency, and inclusive, stakeholder-driven approaches to reduce harm in real-world deployments.

Abstract

Building equitable and inclusive NLP technologies demands consideration of whether and how social attitudes are represented in ML models. In particular, representations encoded in models often inadvertently perpetuate undesirable social biases from the data on which they are trained. In this paper, we present evidence of such undesirable biases towards mentions of disability in two different English language models: toxicity prediction and sentiment analysis. Next, we demonstrate that the neural embeddings that are the critical first step in most NLP pipelines similarly contain undesirable biases towards mentions of disability. We end by highlighting topical biases in the discourse about disability which may contribute to the observed model biases; for instance, gun violence, homelessness, and drug addiction are over-represented in texts discussing mental illness.

Paper Structure

This paper contains 12 sections, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Linguistic Phrases for Disabilities
  3. Biases in Text Classification Models
  4. Biases in Language Representations
  5. Biases in Data
  6. Implications of Model Biases
  7. Discussion and Conclusion
  8. Appendices
  9. Expressions for Disability
  10. Tabular versions of results
  11. Text classification analyses for individual phrases
  12. Additional details of BERT analysis

Figures (5)

  • Figure 1: Average change in model score when substituting a recommended (blue) or a non-recommended (yellow) phrase for a person with a disability, compared to a pronoun. Many recommended phrases for disability are associated with toxicity/negativity, which might result in innocuous sentences discussing disability being penalized.
  • Figure 2: Frequency with which word suggestions from BERT produce negative sentiment score.
  • Figure 3: Average change in toxicity model score when substituting each phrase, compared to using a pronoun
  • Figure 4: Average change in sentiment model score when substituting each phrase, compared to using a pronoun
  • Figure 5: Words produced by BERT in the fill-in-the-blank analysis for sentences containing disability terms that produced negative sentiment scores. Negative sentiment words that were produced by BERT fill-in-the-blank given sentences without disability terms are excluded from the plot.