Language Models Largely Exhibit Human-like Constituent Ordering Preferences

TL;DR

The paper investigates how post-verbal constituent movement in English is handled by a broad set of language-processing systems across four shift types (heavy NP shift, particle movement, dative alternation, multiple PP shift). It combines synthetic and mined data with a log-probability (surprisal) approach and GAMM analyses to assess weight-based predictors (e.g., syllable weight, word length, token length, modifiers) of shifting and to compare model preferences with human judgments. The key finding is that, overall, large language models exhibit human-like constituent ordering preferences, with strong alignment for several shift types, though particle movement remains a notable exception and instruction-tuned variants sometimes align less well with humans. The results support weight-based theories of constituent ordering in both humans and learning systems, highlight syllable weight as a particularly predictive metric, and point to nuanced effects of instruction-tuning on linguistic preferences, suggesting directions for future cross-linguistic and generation-focused research.

Abstract

Though English sentences are typically inflexible vis-à-vis word order, constituents often show far more variability in ordering. One prominent theory presents the notion that constituent ordering is directly correlated with constituent weight: a measure of the constituent's length or complexity. Such theories are interesting in the context of natural language processing (NLP), because while recent advances in NLP have led to significant gains in the performance of large language models (LLMs), much remains unclear about how these models process language, and how this compares to human language processing. In particular, the question remains whether LLMs display the same patterns with constituent movement, and may provide insights into existing theories on when and how the shift occurs in human language. We compare a variety of LLMs with diverse properties to evaluate broad LLM performance on four types of constituent movement: heavy NP shift, particle movement, dative alternation, and multiple PPs. Despite performing unexpectedly around particle movement, LLMs generally align with human preferences around constituent ordering.

Figures (62)

• Figure 1: Examples of constituent movement types: Heavy NP Shift (HNPS), Particle Movement (PM), Dative Alternation (DA) and Multiple PP Shift (MPP).
• Figure 2: We categorize our work into three main experiments. Our first experiment evaluates model response to constituent movement, our second experiment analyzes what motivates LLM constituent ordering preferences, and our third experiment compares model preferences with human judgements.
• Figure 3: Example from pinker2007language.
• Figure 4: The outline for creating synthetic data, using varying modifier weights.
• Figure 5: OLMo 7B M$_{preference}$ scores with respect to different measures of weight.