Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Which Word Orders Facilitate Length Generalization in LMs? An Investigation with GCG-Based Artificial Languages

Nadine El-Naggar, Tatsuki Kuribayashi, Ted Briscoe

TL;DR

The paper introduces a GCG-based artificial-language framework to study how language models generalize to longer sentences and whether typologically common word orders facilitate productive learning. By constructing 96 ALs across varied word orders and testing three architectures (RNN, LSTM, Transformer) on short training data with longer generalization tasks, the work demonstrates that typologically plausible orders tend to be easier to generalize, with recurrent models aligning more closely to typology than Transformers. The study highlights that out-of-domain evaluation reveals clearer inductive biases than in-domain tests, and that length-generalization performance correlates with typological frequency, particularly for RNNs. It also shows that targeted grammaticality judgments corroborate the notion that typology influences grammatical competence, though effects can be construction- and model-dependent. Overall, the findings suggest working-memory constraints shape typological preferences in LMs and provide a framework for evaluating length-generalization beyond standard perplexity metrics.

Abstract

Whether language models (LMs) have inductive biases that favor typologically frequent grammatical properties over rare, implausible ones has been investigated, typically using artificial languages (ALs) (White and Cotterell, 2021; Kuribayashi et al., 2024). In this paper, we extend these works from two perspectives. First, we extend their context-free AL formalization by adopting Generalized Categorial Grammar (GCG) (Wood, 2014), which allows ALs to cover attested but previously overlooked constructions, such as unbounded dependency and mildly context-sensitive structures. Second, our evaluation focuses more on the generalization ability of LMs to process unseen longer test sentences. Thus, our ALs better capture features of natural languages and our experimental paradigm leads to clearer conclusions -- typologically plausible word orders tend to be easier for LMs to productively generalize.

Which Word Orders Facilitate Length Generalization in LMs? An Investigation with GCG-Based Artificial Languages

TL;DR

The paper introduces a GCG-based artificial-language framework to study how language models generalize to longer sentences and whether typologically common word orders facilitate productive learning. By constructing 96 ALs across varied word orders and testing three architectures (RNN, LSTM, Transformer) on short training data with longer generalization tasks, the work demonstrates that typologically plausible orders tend to be easier to generalize, with recurrent models aligning more closely to typology than Transformers. The study highlights that out-of-domain evaluation reveals clearer inductive biases than in-domain tests, and that length-generalization performance correlates with typological frequency, particularly for RNNs. It also shows that targeted grammaticality judgments corroborate the notion that typology influences grammatical competence, though effects can be construction- and model-dependent. Overall, the findings suggest working-memory constraints shape typological preferences in LMs and provide a framework for evaluating length-generalization beyond standard perplexity metrics.

Abstract

Whether language models (LMs) have inductive biases that favor typologically frequent grammatical properties over rare, implausible ones has been investigated, typically using artificial languages (ALs) (White and Cotterell, 2021; Kuribayashi et al., 2024). In this paper, we extend these works from two perspectives. First, we extend their context-free AL formalization by adopting Generalized Categorial Grammar (GCG) (Wood, 2014), which allows ALs to cover attested but previously overlooked constructions, such as unbounded dependency and mildly context-sensitive structures. Second, our evaluation focuses more on the generalization ability of LMs to process unseen longer test sentences. Thus, our ALs better capture features of natural languages and our experimental paradigm leads to clearer conclusions -- typologically plausible word orders tend to be easier for LMs to productively generalize.
Paper Structure (30 sections, 1 equation, 2 figures, 13 tables)

This paper contains 30 sections, 1 equation, 2 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Artificial Language Learning
  4. Generalization to Longer Sentences
  5. Categorial Grammar
  6. Dataset
  7. Overview
  8. Lexicon
  9. Generating the Datasets
  10. Experimental Settings
  11. Models.
  12. Training.
  13. Evaluation.
  14. Typological alignment (TA).
  15. Experiment 1: PPLs
  16. ...and 15 more sections

Figures (2)

  • Figure 1: Example of a sentence and its derivation.
  • Figure 2: Distributions of perplexities and typological plausibility across languages. The error bars indicate max and min PPLs within three runs.