CxMP: A Linguistic Minimal-Pair Benchmark for Evaluating Constructional Understanding in Language Models

Abstract

Recent work has examined language models from a linguistic perspective to better understand how they acquire language. Most existing benchmarks focus on judging grammatical acceptability, whereas the ability to interpret meanings conveyed by grammatical forms has received much less attention. We introduce the Linguistic Minimal-Pair Benchmark for Evaluating Constructional Understanding in Language Models (CxMP), a benchmark grounded in Construction Grammar that treats form-meaning pairings, or constructions, as fundamental linguistic units. CxMP evaluates whether models can interpret the semantic relations implied by constructions, using a controlled minimal-pair design across nine construction types, including the let-alone, caused motion, and ditransitive constructions. Our results show that while syntactic competence emerges early, constructional understanding develops more gradually and remains limited even in large language models (LLMs). CxMP thus reveals persistent gaps in how language models integrate form and meaning, providing a framework for studying constructional understanding and learning trajectories in language models.

Figures (8)

• Figure 1: Overview of the let-alone construction. Humans can grasp the relation between the two entities from sentences containing this construction.
• Figure 2: Scores of each language model across constructions and variants. Solid and dashed lines at the top of the figure represent the scores of GPT-5 for variants A and B, respectively.
• Figure 3: Scores of Pythia models by model size.
• Figure 4: Scores of open-sci-ref models by data size.
• Figure 5: Scores of Llama3.1 base/instruction models.