Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varieties

TL;DR

Trans-EnV presents an automatic framework to transform SAE benchmarks into 38 English varieties (18 dialects and 20 ESL varieties) by blending expert linguistic resources with LLM-based generation. The approach constructs feature-specific transformation guidelines, applies them with a feature transformer and semantic checker to preserve meaning, and validates transformations with human experts. Across six benchmark datasets and seven LLMs, non-SAE varieties impose substantial performance degradation, especially ESL English, highlighting the need for systematic linguistic robustness evaluation. Publicly available code and data accompany the framework, and the work discusses limitations and broader impacts to support responsible deployment.

Abstract

Large Language Models (LLMs) are predominantly evaluated on Standard American English (SAE), often overlooking the diversity of global English varieties. This narrow focus may raise fairness concerns as degraded performance on non-standard varieties can lead to unequal benefits for users worldwide. Therefore, it is critical to extensively evaluate the linguistic robustness of LLMs on multiple non-standard English varieties. We introduce Trans-EnV, a framework that automatically transforms SAE datasets into multiple English varieties to evaluate the linguistic robustness. Our framework combines (1) linguistics expert knowledge to curate variety-specific features and transformation guidelines from linguistic literature and corpora, and (2) LLM-based transformations to ensure both linguistic validity and scalability. Using Trans-EnV, we transform six benchmark datasets into 38 English varieties and evaluate seven state-of-the-art LLMs. Our results reveal significant performance disparities, with accuracy decreasing by up to 46.3% on non-standard varieties. These findings highlight the importance of comprehensive linguistic robustness evaluation across diverse English varieties. Each construction of Trans-EnV was validated through rigorous statistical testing and consultation with a researcher in the field of second language acquisition, ensuring its linguistic validity. Our code and datasets are publicly available at https://github.com/jiyounglee-0523/TransEnV and https://huggingface.co/collections/jiyounglee0523/transenv-681eadb3c0c8cf363b363fb1.

Figures (6)

• Figure 1: Overview of Trans-EnV. (a) Data Collection and Transformation Guideline Generation: We gather English varieties and their associated linguistic features from linguistic literature and large-scale corpora. For each feature, we construct a transformation guideline that defines the procedure for applying the feature. (b) Transformation into Target Variety: Given an SAE sentence $s$ and a target variety $v_i$, the semantic checker $S$ and feature transformer $T$ LLMs transform $s$ by sequentially applying the features of $v_i$ by following guidelines.
• Figure 2: Correlation between linguistic distance and model performance degradation.
• Figure 3: Histograms of distributions of higher-level word usage in CEFR A and CEFR B texts.
• Figure 4: Interface used for human evaluation.
• Figure 5: Correlation between linguistic distance and model performance degradation.