Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Targeted Syntactic Evaluation of Language Models on Georgian Case Alignment

Daniel Gallagher, Gerhard Heyer

TL;DR

This work targets the targeted syntactic evaluation of transformer-based language processing on Georgian split-ergativity, using a Grew-driven minimal-pair test suite derived from the UD Georgian treebank. It introduces a dataset of 370 tests across seven tasks to probe nominative, dative, and ergative marking and evaluates seven language processing systems with word-level and sentence-level metrics. Results show robust nominative performance but systematic struggles with ergative marking, with accuracy correlating with case frequency in Georgian ($Nom > Dat > Erg$) and with data scarcity. The data-driven approach and public dataset provide a transferable framework for evaluating rare syntactic phenomena in low-resource languages and can guide future cross-language studies.

Abstract

This paper evaluates the performance of transformer-based language models on split-ergative case alignment in Georgian, a particularly rare system for assigning grammatical cases to mark argument roles. We focus on subject and object marking determined through various permutations of nominative, ergative, and dative noun forms. A treebank-based approach for the generation of minimal pairs using the Grew query language is implemented. We create a dataset of 370 syntactic tests made up of seven tasks containing 50-70 samples each, where three noun forms are tested in any given sample. Five encoder- and two decoder-only models are evaluated with word- and/or sentence-level accuracy metrics. Regardless of the specific syntactic makeup, models performed worst in assigning the ergative case correctly and strongest in assigning the nominative case correctly. Performance correlated with the overall frequency distribution of the three forms (NOM > DAT > ERG). Though data scarcity is a known issue for low-resource languages, we show that the highly specific role of the ergative along with a lack of available training data likely contributes to poor performance on this case. The dataset is made publicly available and the methodology provides an interesting avenue for future syntactic evaluations of languages where benchmarks are limited.

Targeted Syntactic Evaluation of Language Models on Georgian Case Alignment

TL;DR

This work targets the targeted syntactic evaluation of transformer-based language processing on Georgian split-ergativity, using a Grew-driven minimal-pair test suite derived from the UD Georgian treebank. It introduces a dataset of 370 tests across seven tasks to probe nominative, dative, and ergative marking and evaluates seven language processing systems with word-level and sentence-level metrics. Results show robust nominative performance but systematic struggles with ergative marking, with accuracy correlating with case frequency in Georgian () and with data scarcity. The data-driven approach and public dataset provide a transferable framework for evaluating rare syntactic phenomena in low-resource languages and can guide future cross-language studies.

Abstract

This paper evaluates the performance of transformer-based language models on split-ergative case alignment in Georgian, a particularly rare system for assigning grammatical cases to mark argument roles. We focus on subject and object marking determined through various permutations of nominative, ergative, and dative noun forms. A treebank-based approach for the generation of minimal pairs using the Grew query language is implemented. We create a dataset of 370 syntactic tests made up of seven tasks containing 50-70 samples each, where three noun forms are tested in any given sample. Five encoder- and two decoder-only models are evaluated with word- and/or sentence-level accuracy metrics. Regardless of the specific syntactic makeup, models performed worst in assigning the ergative case correctly and strongest in assigning the nominative case correctly. Performance correlated with the overall frequency distribution of the three forms (NOM > DAT > ERG). Though data scarcity is a known issue for low-resource languages, we show that the highly specific role of the ergative along with a lack of available training data likely contributes to poor performance on this case. The dataset is made publicly available and the methodology provides an interesting avenue for future syntactic evaluations of languages where benchmarks are limited.
Paper Structure (20 sections, 1 equation, 3 figures, 3 tables)

This paper contains 20 sections, 1 equation, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Linguistic Minimal Pairs
  4. Low-Resource Languages
  5. Georgian Syntax
  6. Accusativity & Ergativity
  7. Split Ergativity
  8. Data
  9. Methodology
  10. Minimal Sets
  11. Metrics
  12. Word- versus Sentence-level
  13. Why use one or the other?
  14. Experiments
  15. Results
  16. ...and 5 more sections

Figures (3)

  • Figure 1: The Georgian split-ergative case system, made up of accusative, ergative, and inverted alignments. Examples are provided top-to-bottom for intransitive and transitive constructions. In most languages, including English, there is solely accusative alignment.
  • Figure 2: Grew queries to match sentences in the GLC UD treebank that are transitive erg-nom and intransitive nom constructions.
  • Figure 3: The average word-level probability assigned to each case represented as box plots. In all cases, the highest average probability is assigned to the correct grammatical case. Most significantly however, the average probability is significantly lower where the model must assign the ergative case correctly.