DialectGen: Benchmarking and Improving Dialect Robustness in Multimodal Generation

TL;DR

DialectGen addresses the challenge of dialect robustness in multimodal generation by building a six-dialect benchmark and evaluating 17 models, revealing substantial performance drops when dialect lexemes appear in prompts. The authors design an encoder-based mitigation combining Dialect Learning, Polysemy Control, and KL regularization, achieving gains across five dialects while preserving SAE performance (e.g., SD1.5: ~77.68% dialect performance versus SAE 77.91%). The work provides a rigorous evaluation framework with automatic (VQAScore/CLIPScore) and human validation, plus a reproducible methodology and dataset to standardize future research on dialectal robustness in text-to-image and text-to-video models. The findings underscore the importance of lexical variation handling and offer a practical path toward fairer, dialect-aware multimodal systems.

Abstract

Contact languages like English exhibit rich regional variations in the form of dialects, which are often used by dialect speakers interacting with generative models. However, can multimodal generative models effectively produce content given dialectal textual input? In this work, we study this question by constructing a new large-scale benchmark spanning six common English dialects. We work with dialect speakers to collect and verify over 4200 unique prompts and evaluate on 17 image and video generative models. Our automatic and human evaluation results show that current state-of-the-art multimodal generative models exhibit 32.26% to 48.17% performance degradation when a single dialect word is used in the prompt. Common mitigation methods such as fine-tuning and prompt rewriting can only improve dialect performance by small margins (< 7%), while potentially incurring significant performance degradation in Standard American English (SAE). To this end, we design a general encoder-based mitigation strategy for multimodal generative models. Our method teaches the model to recognize new dialect features while preserving SAE performance. Experiments on models such as Stable Diffusion 1.5 show that our method is able to simultaneously raise performance on five dialects to be on par with SAE (+34.4%), while incurring near zero cost to SAE performance.

Figures (5)

• Figure 1: Multimodal Generative Model Outputs on semantically identical prompts that differ only in one synonymous lexical feature in Standard American English (top) / a lower-resource English dialect (bottom).
• Figure 2: Losses used in our mitigation. Text prompts for Dialect Learning and Polysemy Control come from the DialectGen training set, while image-caption pairs for KL Regularization come from the MSCOCO validation set.
• Figure 3: Qualitative Comparison of Mitigation Strategies using the Stable Diffusion 1.5 model rombach2022high on four different dialect prompts. Specifically, we compare the dialect prompt image generation results of the Stable Diffusion 1.5 Base Model, Stable Diffusion 1.5 fine-tuned with Diffusion DPO wallace2024diffusion, and Stable Diffusion 1.5 updated via our best performing method (Dialect Learning + Image KL Regularization + Polysemy Control).
• Figure 4: The Amazon Mechanical Turk Data Annotation Interface for dialect speaker human filtering of generated prompts (prompt generation details in \ref{['sec:dataset']}). Human annotators may use the "View Instructions" button to collapse / re-open detailed annotation instructions at any time. The annotation interface places no maximum time limit on each annotation question. Human annotators are allowed to return to previously annotated questions and update their answers at any time.
• Figure 5: The English Dialect Speaker Assessment Quiz used for matching dialect speaker annotators to specific dialects for prompt annotation. We adapt the assessment quiz from the existing English Dialect Speaker Survey first created in MultiVALUE Ziems2023MultiVALUEAF, which asks the human annotator to select their linguistic acceptability preference for 10 different dialect excerpts.