Can Language Models Learn Typologically Implausible Languages?

TL;DR

This work investigates whether language models exhibit typologically aligned learning biases by training autoregressive and masked LMs on counterfactual English (head-initial) and Japanese (head-final) corpora that systematically violate Greenbergian correlations. Using a top-down, naturalistic data manipulation pipeline, the authors generate multiple counterfactual variants and evaluate learnability via perplexity trajectories, minimal-pair preferences, and BLiMP/JBLiMP benchmarks in a symmetrical cross-lingual design. The findings indicate LMs are often slower to learn implausible languages, and while final performance is mixed, there is a consistent bias toward harmonic word orders, supporting the view that typological patterns can arise from domain-general learning biases. These results contribute to the human-language typology debate by suggesting that language-specific biases may not be strictly necessary to explain typological distributions, and they demonstrate how LMs can serve as scalable, naturalistic probes of language universals and their cognitive underpinnings.

Abstract

Grammatical features across human languages show intriguing correlations often attributed to learning biases in humans. However, empirical evidence has been limited to experiments with highly simplified artificial languages, and whether these correlations arise from domain-general or language-specific biases remains a matter of debate. Language models (LMs) provide an opportunity to study artificial language learning at a large scale and with a high degree of naturalism. In this paper, we begin with an in-depth discussion of how LMs allow us to better determine the role of domain-general learning biases in language universals. We then assess learnability differences for LMs resulting from typologically plausible and implausible languages closely following the word-order universals identified by linguistic typologists. We conduct a symmetrical cross-lingual study training and testing LMs on an array of highly naturalistic but counterfactual versions of the English (head-initial) and Japanese (head-final) languages. Compared to similar work, our datasets are more naturalistic and fall closer to the boundary of plausibility. Our experiments show that these LMs are often slower to learn these subtly implausible languages, while ultimately achieving similar performance on some metrics regardless of typological plausibility. These findings lend credence to the conclusion that LMs do show some typologically-aligned learning preferences, and that the typological patterns may result from, at least to some degree, domain-general learning biases.

Figures (5)

• Figure 1: Histogram of the number of swaps per sentence for each counterfactual language.
• Figure 2: Final PPLs on respective heldout data for English-based counterfactual LMs (left) and Japanese-based counterfactual LMs (right). Error bars indicate standard deviation over three random seeds.
• Figure 3: Performance trajectories for minimal pair comparisons targeting the counterfactual word order for counterfactual models and natural order for baseline model, for English-based LMs (left) and Japanese-based LMs (right). Shaded areas present standard deviation (SD) over three random seeds.
• Figure 4: Performance trajectories of English-based counterfactual LMs in BLiMP (left) and Japanese-based counterfactual LMs in JBLiMP (right). Shaded areas present SDs over three random seeds.
• Figure 5: Illustration of different levels of tightness when classifying verbal dependents as objects.