Readability $\ne$ Learnability: Rethinking the Role of Simplicity in Training Small Language Models

TL;DR

Challenging the link between human readability and learnability, the paper shows that small language models achieve coherence from data that is statistically simple rather than human-understandable. By constructing matched-structure datasets that vary readability, the authors demonstrate that adult-level text can yield comparable or faster coherence development, while learnability tracks with low n-gram diversity ($n$-gram diversity across $n$ = 1 to 8) rather than readability. The work highlights the brittleness of SLMs trained on highly simplified text and cautions against anthropomorphic interpretations of model learning, advocating for a distributional, statistically grounded view of learnability. It proposes directions for richer measures of dataset complexity and training curricula that preserve generalization while exploiting statistical simplicity.

Abstract

Recent studies suggest that very small language models (SLMs) can generate surprisingly coherent text when trained on simplified, child-directed corpora such as TinyStories. These findings have been interpreted as evidence that readability -- characterized by accessible vocabulary, familiar narrative structure, and simple syntax -- plays a key role in enabling such capabilities to emerge. In this paper, we challenge that interpretation. We construct synthetic datasets with matched structure but varied readability, and find that readability alone does not predict coherence or learning efficiency in SLMs. Models trained on complex, adult-level text perform comparably to those trained on simplified language, and even exhibit faster development of coherence during training. Instead, we show that statistical simplicity, as measured by n-gram diversity, is a stronger predictor of learnability. Our findings caution against the growing trend of anthropomorphizing language model training -- drawing parallels to human cognitive development without empirical basis -- and argue for more precise reasoning about what properties actually support capability emergence in small models.

Figures (37)

• Figure 1: Validation of readability and quality metrics. (a) LLM-based readability scores correlate best with human judgments ($r = 0.74$ via Llama-3.1-70B-Instruct on CLEAR dataset), outperforming classic formulas and parsing metrics. (b) LLM-based coherence scores correlate best with our model ranking (Table \ref{['tab:model_rankings']}), outperforming perplexity.
• Figure 2: Unique n-gram counts across datasets (100M token samples). The synthetic datasets (TinyStories, LlamaTales series) are statistically simpler than standard pretraining corpora (FineWeb, Dolma, SlimPajama), as demonstrated by their substantially fewer unique n-grams (indicating lower diversity and higher predictability).
• Figure 3: LLM-judged coherence scores versus model size. Colors indicate training data; black indicates public reference models. High in-distribution coherence is achievable regardless of readability, alongside strong distribution dependence. Red line: training data coherence for the respective prompt set. See Figure \ref{['fig:quality_coherence_supp']} for results for FineWeb and TinyStories.
• Figure 4: Statistical simplicity predicts learnability; readability does not determine learning speed. (a) Learnability ratio (output coherence / train data coherence) vs unique 3-grams, showing higher learnability for statistically simpler datasets (top-left). (b) Coherence during training. Note the rapid initial coherence emergence when training on less readable LlamaTales-GRE (blue) compared to high-readability datasets (green/orange).
• Figure 5: N-gram novelty (percentage of generated n-grams absent in the training data) vs. n-gram size for SLMs trained on synthetic datasets. Substantial novelty indicates recombination beyond memorization. Llama-3.1-70B (black dots), serving as a baseline, was not trained on these datasets; its generations are compared against each training set here.