Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Differences in Typological Alignment in Language Models' Treatment of Differential Argument Marking

Iskar Deng, Nathalia Xu, Shane Steinert-Threlkeld

TL;DR

This paper trains GPT-2 models on 18 corpora implementing distinct DAM systems and evaluates their generalization using minimal pairs, revealing a dissociation between two typological dimensions of DAM.

Abstract

Recent work has shown that language models (LMs) trained on synthetic corpora can exhibit typological preferences that resemble cross-linguistic regularities in human languages, particularly for syntactic phenomena such as word order. In this paper, we extend this paradigm to differential argument marking (DAM), a semantic licensing system in which morphological marking depends on semantic prominence. Using a controlled synthetic learning method, we train GPT-2 models on 18 corpora implementing distinct DAM systems and evaluate their generalization using minimal pairs. Our results reveal a dissociation between two typological dimensions of DAM. Models reliably exhibit human-like preferences for natural markedness direction, favoring systems in which overt marking targets semantically atypical arguments. In contrast, models do not reproduce the strong object preference in human languages, in which overt marking in DAM more often targets objects rather than subjects. These findings suggest that different typological tendencies may arise from distinct underlying sources.

Differences in Typological Alignment in Language Models' Treatment of Differential Argument Marking

TL;DR

This paper trains GPT-2 models on 18 corpora implementing distinct DAM systems and evaluates their generalization using minimal pairs, revealing a dissociation between two typological dimensions of DAM.

Abstract

Recent work has shown that language models (LMs) trained on synthetic corpora can exhibit typological preferences that resemble cross-linguistic regularities in human languages, particularly for syntactic phenomena such as word order. In this paper, we extend this paradigm to differential argument marking (DAM), a semantic licensing system in which morphological marking depends on semantic prominence. Using a controlled synthetic learning method, we train GPT-2 models on 18 corpora implementing distinct DAM systems and evaluate their generalization using minimal pairs. Our results reveal a dissociation between two typological dimensions of DAM. Models reliably exhibit human-like preferences for natural markedness direction, favoring systems in which overt marking targets semantically atypical arguments. In contrast, models do not reproduce the strong object preference in human languages, in which overt marking in DAM more often targets objects rather than subjects. These findings suggest that different typological tendencies may arise from distinct underlying sources.
Paper Structure (31 sections, 1 equation, 7 figures, 7 tables)

This paper contains 31 sections, 1 equation, 7 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Synthetic Corpus Paradigms and Typological Tendencies in LMs
  4. Differential Argument Marking
  5. Markedness.
  6. Argument Preference.
  7. Synthetic DAM Corpora
  8. Semantic Trigger
  9. Dependency Complexity
  10. Markedness Direction
  11. Argument Target
  12. Experiments
  13. Corpus Construction
  14. Models and Training
  15. Rule Mastery Evaluation
  16. ...and 16 more sections

Figures (7)

  • Figure 1: Overview of the DAM rule injection and the overall results for markedness and argument preference averaged from all DAM rule mastery accuracies.
  • Figure 2: Rule mastery accuracy over training steps for DAM rules.
  • Figure 3: Rule mastery accuracy of the best checkpoint across DAM conditions. We select models based on best validation perplexity.
  • Figure 4: Marker placement accuracy of the best checkpoint across DAM conditions. We select models based on best validation perplexity.
  • Figure 5: Semantic probing accuracy at the best training checkpoint across DAM conditions, evaluated separately for subject and object representations.
  • ...and 2 more figures