Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Hrunting of AI: Where and How to Improve English Dialectal Fairness

Wei Li, Adrian de Wynter

Abstract

It is known that large language models (LLMs) underperform in English dialects, and that improving them is difficult due to data scarcity. In this work we investigate how quality and availability impact the feasibility of improving LLMs in this context. For this, we evaluate three rarely-studied English dialects (Yorkshire, Geordie, and Cornish), plus African-American Vernacular English, and West Frisian as control. We find that human-human agreement when determining LLM generation quality directly impacts LLM-as-a-judge performance. That is, LLM-human agreement mimics the human-human agreement pattern, and so do metrics such as accuracy. It is an issue because LLM-human agreement measures an LLM's alignment with the human consensus; and hence raises questions about the feasibility of improving LLM performance in locales where low populations induce low agreement. We also note that fine-tuning does not eradicate, and might amplify, this pattern in English dialects. But also find encouraging signals, such as some LLMs' ability to generate high-quality data, thus enabling scalability. We argue that data must be carefully evaluated to ensure fair and inclusive LLM improvement; and, in the presence of scarcity, new tools are needed to handle the pattern found.

The Hrunting of AI: Where and How to Improve English Dialectal Fairness

Abstract

It is known that large language models (LLMs) underperform in English dialects, and that improving them is difficult due to data scarcity. In this work we investigate how quality and availability impact the feasibility of improving LLMs in this context. For this, we evaluate three rarely-studied English dialects (Yorkshire, Geordie, and Cornish), plus African-American Vernacular English, and West Frisian as control. We find that human-human agreement when determining LLM generation quality directly impacts LLM-as-a-judge performance. That is, LLM-human agreement mimics the human-human agreement pattern, and so do metrics such as accuracy. It is an issue because LLM-human agreement measures an LLM's alignment with the human consensus; and hence raises questions about the feasibility of improving LLM performance in locales where low populations induce low agreement. We also note that fine-tuning does not eradicate, and might amplify, this pattern in English dialects. But also find encouraging signals, such as some LLMs' ability to generate high-quality data, thus enabling scalability. We argue that data must be carefully evaluated to ensure fair and inclusive LLM improvement; and, in the presence of scarcity, new tools are needed to handle the pattern found.
Paper Structure (61 sections, 8 figures, 15 tables)

This paper contains 61 sections, 8 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Findings
  3. Methods
  4. Languages and Dialects Covered
  5. Approach
  6. Rubric and Annotation Metrics
  7. c1:
  8. c2:
  9. c3:
  10. c4:
  11. c5:
  12. c6:
  13. TSD Collection
  14. Human Annotation
  15. LLMs Used
  16. ...and 46 more sections

Figures (8)

  • Figure 1: Geographical distribution for the English dialects used in this work and our control, West Frisian.
  • Figure 2: Average AC1 between humans (H-H) and humans and LLMs (H-LLM), compared with the LLMs' averaged F$_1$, with the same dataset and (best-performing) ICL prompt. H-H roughly aligns with H-LLM, and with the LLMs' F$_1$. H-LLM, in theory, should be near 1.0, even if H-H is not, to indicate agreement with the human consensus.
  • Figure 3: Levenshtein distance versus LLM performance per locale. A logistic model shows that distance negatively predicts the odds of label=1 in West Frisian, but not in English dialects ($p$ < 0.001).
  • Figure 4: Strategy 4 LLM-human agreements for (left-to-right) AAVE, Cornish, Geordie, Yorkshire, and West Frisian. Top: agreements during ICL (all LLMs). Bottom: agreements for fine-tuned Qwen3. During ICL, LLM-human AC1 approximated the human-human AC1 distribution: high AC1s corresponded to locales with high human-human AC1 (AAVE, Yorkshire), and symmetrically for low AC1 (Cornish, primarily). LLM performance as measured by F$_1$ also followed this pattern. After fine-tuning, AC1 increased, although in dialects these improvements largely followed the pattern. West Frisian had low AC1, but presented noticeable improvements.
  • Figure 5: Breakdown by source of human-determined high quality outputs for (top) AAVE, Cornish, Geordie, and (bottom) Yorkshire and West Frisian. West Frisian and Cornish had outputs considered of very low quality, which, upon further inspection, was due to the model often responding in the wrong locale. On average, the lowest performance was by Qwen3 (51.2%), followed by GPT-5 (60.2%), GPT-4.1 (70.3%), and Opus-4.1 (72.1%). The average AC1 between human annotators was 0.75.
  • ...and 3 more figures