Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Corpus-Grounded Agentic LLMs for Multilingual Grammatical Analysis

Matej Klemen, Tjaša Arčon, Luka Terčon, Marko Robnik-Šikonja, Kaja Dobrovoljc

TL;DR

The paper introduces an agentic LLM framework for corpus-grounded grammatical analysis and demonstrates it on multilingual word-order tasks drawn from UD data with WALS-inspired questions. It formalizes a three-layer output task (dominant order, attested orders, distribution) and evaluates a prototype system (UDagent) against baselines, showing strong performance across 170+ languages and 13 features while highlighting failures due to code-generation and interpretation issues. The work provides a scalable, interpretable approach to combining LLM reasoning with structured linguistic data and outlines clear paths for extending the methodology to broader linguistic phenomena and open-model deployments.

Abstract

Empirical grammar research has become increasingly data-driven, but the systematic analysis of annotated corpora still requires substantial methodological and technical effort. We explore how agentic large language models (LLMs) can streamline this process by reasoning over annotated corpora and producing interpretable, data-grounded answers to linguistic questions. We introduce an agentic framework for corpus-grounded grammatical analysis that integrates concepts such as natural-language task interpretation, code generation, and data-driven reasoning. As a proof of concept, we apply it to Universal Dependencies (UD) corpora, testing it on multilingual grammatical tasks inspired by the World Atlas of Language Structures (WALS). The evaluation spans 13 word-order features and over 170 languages, assessing system performance across three complementary dimensions - dominant-order accuracy, order-coverage completeness, and distributional fidelity - which reflect how well the system generalizes, identifies, and quantifies word-order variations. The results demonstrate the feasibility of combining LLM reasoning with structured linguistic data, offering a first step toward interpretable, scalable automation of corpus-based grammatical inquiry.

Towards Corpus-Grounded Agentic LLMs for Multilingual Grammatical Analysis

TL;DR

The paper introduces an agentic LLM framework for corpus-grounded grammatical analysis and demonstrates it on multilingual word-order tasks drawn from UD data with WALS-inspired questions. It formalizes a three-layer output task (dominant order, attested orders, distribution) and evaluates a prototype system (UDagent) against baselines, showing strong performance across 170+ languages and 13 features while highlighting failures due to code-generation and interpretation issues. The work provides a scalable, interpretable approach to combining LLM reasoning with structured linguistic data and outlines clear paths for extending the methodology to broader linguistic phenomena and open-model deployments.

Abstract

Empirical grammar research has become increasingly data-driven, but the systematic analysis of annotated corpora still requires substantial methodological and technical effort. We explore how agentic large language models (LLMs) can streamline this process by reasoning over annotated corpora and producing interpretable, data-grounded answers to linguistic questions. We introduce an agentic framework for corpus-grounded grammatical analysis that integrates concepts such as natural-language task interpretation, code generation, and data-driven reasoning. As a proof of concept, we apply it to Universal Dependencies (UD) corpora, testing it on multilingual grammatical tasks inspired by the World Atlas of Language Structures (WALS). The evaluation spans 13 word-order features and over 170 languages, assessing system performance across three complementary dimensions - dominant-order accuracy, order-coverage completeness, and distributional fidelity - which reflect how well the system generalizes, identifies, and quantifies word-order variations. The results demonstrate the feasibility of combining LLM reasoning with structured linguistic data, offering a first step toward interpretable, scalable automation of corpus-based grammatical inquiry.

Paper Structure

This paper contains 12 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Task and Evaluation Framework
  4. Task Definition
  5. Evaluation Dataset
  6. Evaluation Metrics
  7. Agentic LLM System
  8. Evaluation and Analysis
  9. Overall Performance
  10. Cross-Linguistic Analysis
  11. Error and Failure Analysis
  12. Conclusion

Figures (3)

  • Figure 1: A simplified representation of the contrast between a traditional approach to corpus-based linguistics research (top), and our abstract solution proposal (Grammatical analysis agent, bottom).
  • Figure 2: The proposed UDagent system implementation, along with a system trace showing how the system would handle the problem of determining the order of the subject, verb, and object in the French language.
  • Figure 3: Side-by-side visualization of the predicted and the ground truth intermediate answer distribution for features 81A, 83A, and 84A for Dutch. On features 83A and 84A minor distribution miscalculations result in the wrong dominant answer prediction.