Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Individual and Combined Effects of English as a Second Language and Typos on LLM Performance

Serena Liu, Yutong Yang, Prisha Sheth, Weixuan Dong, Mingjiao Diao, Xinru Zhu, Nikhil Banga, Oscar Melendez, Arnav Sharma, Minda Zhao, Marina Lin, Mengyu Wang

Abstract

Large language models (LLMs) are used globally, and because much of their training data is in English, they typically perform best on English inputs. As a result, many non-native English speakers interact with them in English as a second language (ESL), and these inputs often contain typographical errors. Prior work has largely studied the effects of ESL variation and typographical errors separately, even though they often co-occur in real-world use. In this study, we use the Trans-EnV framework to transform standard English inputs into eight ESL variants and apply MulTypo to inject typos at three levels: low, moderate, and severe. We find that combining ESL variation and typos generally leads to larger performance drops than either factor alone, though the combined effect is not simply additive. This pattern is clearest on closed-ended tasks, where performance degradation can be characterized more consistently across ESL variants and typo levels, while results on open-ended tasks are more mixed. Overall, these findings suggest that evaluations on clean standard English may overestimate real-world model performance, and that evaluating ESL variation and typographical errors in isolation does not fully capture model behavior in realistic settings.

Individual and Combined Effects of English as a Second Language and Typos on LLM Performance

Abstract

Large language models (LLMs) are used globally, and because much of their training data is in English, they typically perform best on English inputs. As a result, many non-native English speakers interact with them in English as a second language (ESL), and these inputs often contain typographical errors. Prior work has largely studied the effects of ESL variation and typographical errors separately, even though they often co-occur in real-world use. In this study, we use the Trans-EnV framework to transform standard English inputs into eight ESL variants and apply MulTypo to inject typos at three levels: low, moderate, and severe. We find that combining ESL variation and typos generally leads to larger performance drops than either factor alone, though the combined effect is not simply additive. This pattern is clearest on closed-ended tasks, where performance degradation can be characterized more consistently across ESL variants and typo levels, while results on open-ended tasks are more mixed. Overall, these findings suggest that evaluations on clean standard English may overestimate real-world model performance, and that evaluating ESL variation and typographical errors in isolation does not fully capture model behavior in realistic settings.

Paper Structure

This paper contains 27 sections, 2 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. LLMs' Language Specific Ability and User Language Bias
  3. Models for ESL Simulation and Typo injection
  4. Contributions
  5. Methodology
  6. ESL transformation (Trans-EnV)
  7. Typographical perturbation (MulTypo)
  8. Condition Construction
  9. Metrics
  10. Experimental Setup
  11. Datasets
  12. Models
  13. Executions
  14. Results
  15. Close Ended Tasks
  16. ...and 12 more sections

Figures (5)

  • Figure 1: Sample LLM outputs on an MMLU question under three perturbations. ESL (Arabic L1, CEFR-A): subject-verb mismatch, article omission. Typographical: keyboard-adjacent character operations with $\sqrt{\text{length}}$-weighted word sampling.
  • Figure 2: MMLU accuracy percentage organized by language and typo rate, per model.
  • Figure 3: GSM8k accuracy percentage organized by language and typo rate, per model.
  • Figure 4: HellaSwag accuracy percentage organized by language and typo rate, per model.
  • Figure 5: Dimension-wise degradation on MT-Bench under combined ESL and typographical perturbations ($r=0.7$).