Evaluating Robustness of Large Language Models Against Multilingual Typographical Errors

TL;DR

This work addresses the vulnerability of multilingual large language models to real-world typographical errors by introducing MulTypo, a keyboard-aware typo generator that models language-specific typing behavior with a corruption ratio $ au \in [0,1]$ and word-length-based sampling using probabilities proportional to $\sqrt{|w|}$. It conducts a comprehensive robustness evaluation of 18 open-source LLMs across five downstream tasks and three model families, under zero- and few-shot prompting and varying typo rates, revealing that typos degrade performance especially in generative and reasoning tasks, while high-resource languages and Latin scripts tend to be more robust. Instruction tuning improves clean-input performance but often increases brittleness under noise, and increasing demonstrations provides limited gains in robustness. The findings underscore the need for noise-aware multilingual pretraining and evaluation, and the authors provide public code and data to advance research in multilingual robustness to typographical errors.

Abstract

Large language models (LLMs) are increasingly deployed in multilingual, real-world applications with user inputs -- naturally introducing typographical errors (typos). Yet most benchmarks assume clean input, leaving the robustness of LLMs to typos across languages largely underexplored. To address this gap, we introduce MulTypo, a multilingual typo generation algorithm that simulates human-like errors based on language-specific keyboard layouts and typing behavior. We evaluate 18 open-source LLMs across three model families and five downstream tasks spanning language inference, multi-choice question answering, mathematical reasoning, and machine translation tasks. Our results show that typos consistently degrade performance, particularly in generative tasks and those requiring reasoning -- while the natural language inference task is comparatively more robust. Instruction tuning improves clean-input performance but may increase brittleness under noise. We also observe language-dependent robustness: high-resource languages are generally more robust than low-resource ones, and translation from English is more robust than translation into English. Our findings underscore the need for noise-aware training and multilingual robustness evaluation. We make our code and data publicly available.

Figures (14)

• Figure 1: Illustration of the impact of real-world typographical errors. Humans often make typos on language-specific keyboard layouts, and once such errors are introduced, models can fail across languages. In this example, the model cannot generate the correct answer ("500") under typos in English, German, and Russian.
• Figure 2: Illustration of the pipeline of MulTypo: Given an input text and a user-defined typo rate, the algorithm (i) samples words with probability proportional to the square root of the word length, (ii) samples character positions using a position-aware distribution, and (iii) samples one of four typo operations: replace, insert, delete, or transpose. Then the algorithm produces a noised text that simulates human-like errors.
• Figure 3: Performance under different typo rates ($0$, $0.1$, $0.4$, and, $0.7$) averaged across languages for each task.
• Figure 4: Impact of model size (Small, Medium, Large) on multilingual robustness across five tasks. A different color represents each model family, and two lines are plotted per family: performance on clean input (0%) and input with a 10% typo rate. Larger models generally perform better but also exhibit performance drops under noise.
• Figure 5: Impact of Instruction-tuning on multilingual robustness. Instruction-tuned models improve the performance, but do not seem to improve the robustness against typos, especially with higher typo rates.