Should LLMs, $\textit{like}$, Generate How Users Talk? Building Dialect-Accurate Dialog[ue]s Beyond the American Default with MDial

TL;DR

This work tackles the dominance of Standard American English in LLMs by introducing MDial, a linguist-validated framework for generating parallel multi-turn dialogs across nine dialects along lexical, orthographic, and morphosyntactic dimensions. It distinguishes between features speakers use and features models should produce, enabling more authentic and dialect-aware interactions. The authors present MDialBench to benchmark dialect identification and dialect-appropriate response generation, revealing substantial gaps even in frontier models and showing that post-training on MDial data yields significant gains. The approach offers a scalable pathway to dialect equity in conversational AI and lays groundwork for future dialect-aware training and evaluation.

Abstract

More than 80% of the 1.6 billion English speakers do not use Standard American English (SAE) and experience higher failure rates and stereotyped responses when interacting with LLMs as a result. Yet multi-dialectal performance remains underexplored. We introduce $\textbf{MDial}$, the first large-scale framework for generating multi-dialectal conversational data encompassing the three pillars of written dialect -- lexical (vocabulary), orthographic (spelling), and morphosyntactic (grammar) features -- for nine English dialects. Partnering with native linguists, we design an annotated and scalable rule-based LLM transformation to ensure precision. Our approach challenges the assumption that models should mirror users' morphosyntactic features, showing that up to 90% of the grammatical features of a dialect should not be reproduced by models. Independent evaluations confirm data quality, with annotators preferring MDial outputs over prior methods in 98% of pairwise comparisons for dialect naturalness. Using this pipeline, we construct the dialect-parallel $\textbf{MDialBench}$mark with 50k+ dialogs, resulting in 97k+ QA pairs, and evaluate 17 LLMs on dialect identification and response generation tasks. Even frontier models achieve under 70% accuracy, fail to reach 50% for Canadian English, and systematically misclassify non-SAE dialects as American or British. As dialect identification underpins natural language understanding, these errors risk cascading failures into downstream tasks.

Figures (26)

• Figure 1: Overview of MDial. The framework combines ortho-lexical seed creation, multi-turn SAE dialog generation, and rule-guided dialect transformation using curated lexical, orthographic, and morphosyntactic knowledge to produce parallel dialogs across 9 English dialects. The resulting dataset supports both benchmarking (MDialBench) and post-training to enhance LLMs' dialectal capabilities.
• Figure 2: Annotation rating distribution of eWAVE, User, and Model appropriate generation for AU morphosyntax features.
• Figure 3: Model performance on MDialBench classification (Identification) task for turn 1 and 8 dialogs. We observe an overall trend of positive correlation between model size & accuracy and that models struggle when morphosyntactic features are added. Results across all turns for classification and generation are shown in Fig. \ref{['fig:classification']} and Fig. \ref{['fig:response_completion']}, respectively.
• Figure 4: Model performance by scale and context length averaged over OrthoLex, $RBT_{User}$, and $RBT_{model}$ data.
• Figure 5: Guidelines provided to Nigerian annotators for evaluating morphosyntactic variations. Examples are truncated for brevity.