Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating the Usage of African-American Vernacular English in Large Language Models

Deja Dunlap, R. Thomas McCoy

TL;DR

This work investigates how accurately large language models (LLMs) represent African American Vernacular English (AAVE) and finds that, in many cases, there are substantial differences between AAVE usage in LLMs and humans.

Abstract

In AI, most evaluations of natural language understanding tasks are conducted in standardized dialects such as Standard American English (SAE). In this work, we investigate how accurately large language models (LLMs) represent African American Vernacular English (AAVE). We analyze three LLMs to compare their usage of AAVE to the usage of humans who natively speak AAVE. We first analyzed interviews from the Corpus of Regional African American Language and TwitterAAE to identify the typical contexts where people use AAVE grammatical features such as ain't. We then prompted the LLMs to produce text in AAVE and compared the model-generated text to human usage patterns. We find that, in many cases, there are substantial differences between AAVE usage in LLMs and humans: LLMs usually underuse and misuse grammatical features characteristic of AAVE. Furthermore, through sentiment analysis and manual inspection, we found that the models replicated stereotypes about African Americans. These results highlight the need for more diversity in training data and the incorporation of fairness methods to mitigate the perpetuation of stereotypes.

Evaluating the Usage of African-American Vernacular English in Large Language Models

TL;DR

This work investigates how accurately large language models (LLMs) represent African American Vernacular English (AAVE) and finds that, in many cases, there are substantial differences between AAVE usage in LLMs and humans.

Abstract

In AI, most evaluations of natural language understanding tasks are conducted in standardized dialects such as Standard American English (SAE). In this work, we investigate how accurately large language models (LLMs) represent African American Vernacular English (AAVE). We analyze three LLMs to compare their usage of AAVE to the usage of humans who natively speak AAVE. We first analyzed interviews from the Corpus of Regional African American Language and TwitterAAE to identify the typical contexts where people use AAVE grammatical features such as ain't. We then prompted the LLMs to produce text in AAVE and compared the model-generated text to human usage patterns. We find that, in many cases, there are substantial differences between AAVE usage in LLMs and humans: LLMs usually underuse and misuse grammatical features characteristic of AAVE. Furthermore, through sentiment analysis and manual inspection, we found that the models replicated stereotypes about African Americans. These results highlight the need for more diversity in training data and the incorporation of fairness methods to mitigate the perpetuation of stereotypes.
Paper Structure (24 sections, 3 figures, 8 tables)

This paper contains 24 sections, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Societal Biases in AI Systems
  4. AAVE in LLMs
  5. Comparing LLM Language to Human Language
  6. Methodology
  7. Human-Generated Language
  8. Model-Generated Text
  9. Experiment 1: Content Analysis
  10. Qualitative analysis of themes
  11. Quantitative analysis of sentiment
  12. Experiment 2: Feature Frequency
  13. Linguistic Phenomena
  14. Grammatical Feature Detection
  15. Statistical Analysis
  16. ...and 9 more sections

Figures (3)

  • Figure 1: The workflow we used to compare how African-American Vernacular English is used by humans and large language models. This workflow is expanded in further detail in the Methodology section below.
  • Figure 2: Bar plots highlighting the portion of sentences in each corpus that had a sentiment score $x$ defined as negative ($x<-0.5$), neutral ($0.5> x >-0.5$), or positive ($x>0.5$), where $x$ was obtained via VADER.
  • Figure 3: Bar plots highlighting the difference in feature frequency by model and feature (per 10,000 sentences). Features that appear less frequently (ie. Double Comparative, Null Modals, Perfective done) have a harder time appearing on scale; see Table \ref{['ref:feature_frequencies']} for those features.