Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Language Models at the Syntax-Semantics Interface: A Case Study of the Long-Distance Binding of Chinese Reflexive ziji

Xiulin Yang

TL;DR

This work investigates whether language models can capture the complex syntax–semantics interface underlying Mandarin Chinese reflexive ziji, particularly its long-distance binding. It introduces a dual data regime (240 synthetic and 320 natural sentences) and evaluates 21 models using perplexity-based metrics, prompting for closed-source LLMs, and native Mandarin human judgments. The study reveals that no model consistently matches human judgments, with models largely relying on linear, sequential cues and showing stronger sensitivity to noun animacy than verb semantics. These findings highlight the limitations of current LMs in abstract binding constraints and motivate refined evaluation methodologies and future work on syntactic–semantic knowledge in multilingual settings.

Abstract

This paper explores whether language models can effectively resolve the complex binding patterns of the Mandarin Chinese reflexive ziji, which are constrained by both syntactic and semantic factors. We construct a dataset of 240 synthetic sentences using templates and examples from syntactic literature, along with 320 natural sentences from the BCC corpus. Evaluating 21 language models against this dataset and comparing their performance to judgments from native Mandarin speakers, we find that none of the models consistently replicates human-like judgments. The results indicate that existing language models tend to rely heavily on sequential cues, though not always favoring the closest strings, and often overlooking subtle semantic and syntactic constraints. They tend to be more sensitive to noun-related than verb-related semantics.

Language Models at the Syntax-Semantics Interface: A Case Study of the Long-Distance Binding of Chinese Reflexive ziji

TL;DR

This work investigates whether language models can capture the complex syntax–semantics interface underlying Mandarin Chinese reflexive ziji, particularly its long-distance binding. It introduces a dual data regime (240 synthetic and 320 natural sentences) and evaluates 21 models using perplexity-based metrics, prompting for closed-source LLMs, and native Mandarin human judgments. The study reveals that no model consistently matches human judgments, with models largely relying on linear, sequential cues and showing stronger sensitivity to noun animacy than verb semantics. These findings highlight the limitations of current LMs in abstract binding constraints and motivate refined evaluation methodologies and future work on syntactic–semantic knowledge in multilingual settings.

Abstract

This paper explores whether language models can effectively resolve the complex binding patterns of the Mandarin Chinese reflexive ziji, which are constrained by both syntactic and semantic factors. We construct a dataset of 240 synthetic sentences using templates and examples from syntactic literature, along with 320 natural sentences from the BCC corpus. Evaluating 21 language models against this dataset and comparing their performance to judgments from native Mandarin speakers, we find that none of the models consistently replicates human-like judgments. The results indicate that existing language models tend to rely heavily on sequential cues, though not always favoring the closest strings, and often overlooking subtle semantic and syntactic constraints. They tend to be more sensitive to noun-related than verb-related semantics.

Paper Structure

This paper contains 28 sections, 2 equations, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Chinese Reflexive ziji
  4. Long-distance Binding
  5. Blocking Effect
  6. Animacy Effect
  7. Subject Orientation
  8. Verb Orientation
  9. Probing Linguistic Knowledge in Language Models
  10. Data
  11. Synthetic Data
  12. In-Context Minimal Pairs
  13. Natural Data
  14. Evaluation
  15. Models
  16. ...and 13 more sections

Figures (3)

  • Figure 1: Examples of binding, the words highlighted in the same color are co-indexed.
  • Figure 2: Local binding tendency caused by the blocking effect based on the baseline result.
  • Figure 3: Accuracy of language models across two settings of the animacy effect: (1) matrix subject is animate and subordinate subject is inanimate (animate < inanimate), and (2) matrix subject is inanimate and subordinate subject is animate (inanimate < animate).