Word Form Matters: LLMs' Semantic Reconstruction under Typoglycemia

TL;DR

This study investigates how large language models reconstruct word meaning under Typoglycemia-style scrambling, asking whether word form or context primarily drives semantic recovery. It introduces SemRecScore, defined as the cosine similarity between the original word representation $x^{(L)}_o$ and the last scrambled subword representation $x^{(L)}_s$ at layer $L$, via $SemRecScore^{(L)}=\frac{x^{(L)}_o\cdot x^{(L)}_s}{\|x^{(L)}_o\|\|x^{(L)}_s\|}$, and validates it against NegCorrRate to link semantic reconstruction with completion-consistency across Scramble Ratios and Context Integrities on LLaMA-3.2/3.3-Instruct. The results show word form is the dominant factor for reconstruction, with context contributing minimally, and reveal a structured, form-focused attention mechanism featuring specialized form-sensitive heads that intensify as scrambling increases. These findings highlight a fundamental difference between human adaptive strategies and fixed attention patterns in LLMs, offering implications for augmenting LLM robustness by incorporating human-like, context-aware processing. The work provides a principled framework for probing internal representations under perturbations and informs future directions across architectures, perturbations, and languages to enhance semantic adaptability.

Abstract

Human readers can efficiently comprehend scrambled words, a phenomenon known as Typoglycemia, primarily by relying on word form; if word form alone is insufficient, they further utilize contextual cues for interpretation. While advanced large language models (LLMs) exhibit similar abilities, the underlying mechanisms remain unclear. To investigate this, we conduct controlled experiments to analyze the roles of word form and contextual information in semantic reconstruction and examine LLM attention patterns. Specifically, we first propose SemRecScore, a reliable metric to quantify the degree of semantic reconstruction, and validate its effectiveness. Using this metric, we study how word form and contextual information influence LLMs' semantic reconstruction ability, identifying word form as the core factor in this process. Furthermore, we analyze how LLMs utilize word form and find that they rely on specialized attention heads to extract and process word form information, with this mechanism remaining stable across varying levels of word scrambling. This distinction between LLMs' fixed attention patterns primarily focused on word form and human readers' adaptive strategy in balancing word form and contextual information provides insights into enhancing LLM performance by incorporating human-like, context-aware mechanisms.

Figures (10)

• Figure 1: Relationship between $\Delta$SR and Average NegCorrScore across LLaMA models of different scales. The increasing trend of NegCorrScore with $\Delta$SR validates SemRecScore as a reliable measure of semantic reconstruction.
• Figure 2: Semantic reconstruction performance across different Scramble Ratios (SR) and Context Integrity (CI) levels. The top row (a-d) presents SemRecScore trends under varying SR values for 1B, 3B, and 70B models. The bottom row (e-h) illustrates SemRecScore evolution for fixed SR values while varying CI. Across all models, word form plays a dominant role, with context integrity having minimal impact on reconstruction performance.
• Figure 3: Attention allocation to word form under varying Scramble Ratios (SR). Subplots (a-c) show AttentionSelf trends for 1B, 3B, and 70B models with full context (CI=1), while (d) presents the 3B model without context (CI=0). Higher SR values consistently elicit stronger attention to word form, and the cyclic attention pattern remains unchanged even without context, suggesting that LLMs process word form independently of contextual information.
• Figure 4: Heatmaps of attention allocation to word form in the LLaMA-1B-Instruct across Scramble Ratios (SR). The x-axis denotes attention heads, and the y-axis denotes layers. Specific heads consistently focus on word form, with higher SR activating more form-sensitive heads, indicating a structured and stable processing mechanism.
• Figure 5: Semantic Reconstruction Performance across Different LLM Scales and Context Integrity Levels. The plots illustrate the layer-wise Semantic Reconstruction Score (SemRecScore) for various SR values across different LLaMA models (1B, 3B, and 70B). The top row represents CI = 0, while the bottom row represents CI = 0.25. The legend indicates different SR conditions, including the “Completely Scrambled” setting. The similarity of the curves across different CI values suggests that Context Integrity (CI) has minimal impact on semantic reconstruction performance.