Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Typoglycemia under the Hood: Investigating Language Models' Understanding of Scrambled Words

Gianluca Sperduti, Alejandro Moreo

TL;DR

This paper investigates why language models tolerate scrambled word interiors (typoglycemia) by combining corpus analysis on the British National Corpus, a BERT-based disambiguation task, and training-from-scratch BERT variants on clean, BE-sorted, and fully sorted text to probe robustness. It finds that word collapse under typical typoglycemia is relatively rare and that contextual cues largely disambiguate collapsed forms, while extreme sorting causes notable information loss; extending training can mitigate some deficits. The work offers a controlled, data-driven account of how distributional signals support reading scrambled text and informs preprocessing and tokenization choices for NLP systems. Overall, the findings suggest that typoglycemia operates within a narrow space of lexical ambiguity and context, enabling near-parallel performance with standard inputs under moderate scrambling while highlighting limits for extreme transformations.

Abstract

Research in linguistics has shown that humans can read words with internally scrambled letters, a phenomenon recently dubbed typoglycemia. Some specific NLP models have recently been proposed that similarly demonstrate robustness to such distortions by ignoring the internal order of characters by design. This raises a fundamental question: how can models perform well when many distinct words (e.g., form and from) collapse into identical representations under typoglycemia? Our work, focusing exclusively on the English language, seeks to shed light on the underlying aspects responsible for this robustness. We hypothesize that the main reasons have to do with the fact that (i) relatively few English words collapse under typoglycemia, and that (ii) collapsed words tend to occur in contexts so distinct that disambiguation becomes trivial. In our analysis, we (i) analyze the British National Corpus to quantify word collapse and ambiguity under typoglycemia, (ii) evaluate BERT's ability to disambiguate collapsing forms, and (iii) conduct a probing experiment by comparing variants of BERT trained from scratch on clean versus typoglycemic Wikipedia text; our results reveal that the performance degradation caused by scrambling is smaller than expected.

Typoglycemia under the Hood: Investigating Language Models' Understanding of Scrambled Words

TL;DR

This paper investigates why language models tolerate scrambled word interiors (typoglycemia) by combining corpus analysis on the British National Corpus, a BERT-based disambiguation task, and training-from-scratch BERT variants on clean, BE-sorted, and fully sorted text to probe robustness. It finds that word collapse under typical typoglycemia is relatively rare and that contextual cues largely disambiguate collapsed forms, while extreme sorting causes notable information loss; extending training can mitigate some deficits. The work offers a controlled, data-driven account of how distributional signals support reading scrambled text and informs preprocessing and tokenization choices for NLP systems. Overall, the findings suggest that typoglycemia operates within a narrow space of lexical ambiguity and context, enabling near-parallel performance with standard inputs under moderate scrambling while highlighting limits for extreme transformations.

Abstract

Research in linguistics has shown that humans can read words with internally scrambled letters, a phenomenon recently dubbed typoglycemia. Some specific NLP models have recently been proposed that similarly demonstrate robustness to such distortions by ignoring the internal order of characters by design. This raises a fundamental question: how can models perform well when many distinct words (e.g., form and from) collapse into identical representations under typoglycemia? Our work, focusing exclusively on the English language, seeks to shed light on the underlying aspects responsible for this robustness. We hypothesize that the main reasons have to do with the fact that (i) relatively few English words collapse under typoglycemia, and that (ii) collapsed words tend to occur in contexts so distinct that disambiguation becomes trivial. In our analysis, we (i) analyze the British National Corpus to quantify word collapse and ambiguity under typoglycemia, (ii) evaluate BERT's ability to disambiguate collapsing forms, and (iii) conduct a probing experiment by comparing variants of BERT trained from scratch on clean versus typoglycemic Wikipedia text; our results reveal that the performance degradation caused by scrambling is smaller than expected.
Paper Structure (20 sections, 3 equations, 5 figures, 13 tables)

This paper contains 20 sections, 3 equations, 5 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. NLP models' robustness to misspellings
  4. Typoglycemia and NLP
  5. Typoglycemia in NLP: Why does it work?
  6. Sorting, sinortg, ginorst: A simple NLP method to explore typoglycemia
  7. This works, but how? The problem of the collapsing effect
  8. Quantifying Collapsing Effects in Typoglycemia
  9. The proof is in the pudding: collapsing words are easy to tell apart using context
  10. Typoglycemic modeling: contextual model can learn from typoglycemic text
  11. Models and Training Details
  12. Tokenizer Construction and Analysis
  13. GLUE Benchmark Evaluation
  14. Does Extended Training Help? Evaluating typoglycemic BERTs with Additional Steps
  15. Conclusion
  16. ...and 5 more sections

Figures (5)

  • Figure 1: Frequency distribution of collapsing words in the classic typoglycemia setting (log scale).
  • Figure 2: Frequency distribution of collapsing words in the extreme typoglycemia setting (log scale).
  • Figure 3: Token frequency distributions (log scale) for the tokenizers in the classic and typoglycemic variants.
  • Figure 4: Growth of the average GLUE score of T-BERT variants as training steps increase.
  • Figure 5: Growth of the average GLUE score of X-BERT variants as training steps increase.