Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prompt and circumstance: A word-by-word LLM prompting approach to interlinear glossing for low-resource languages

Micha Elsner, David Liu

TL;DR

The paper tackles the challenge of efficiently producing interlinear glossed text for low-resource languages by leveraging a retrieval-based, word-by-word prompting approach with a large language model. Using GPT-4o on seven SIGMORPHON 2023 languages, the method beats the Track 1 baseline at the morpheme level and, in many cases, the 3-best oracle matches or exceeds the challenge winner on word-level scores, with a Tsez case study showing that generated linguistic instructions can reduce error rates. The authors further demonstrate that disambiguation instructions for syncretic forms can meaningfully improve gloss accuracy in Tsez, illustrating the potential for human-in-the-loop guidance and linguistically informed prompts. Overall, the work supports the viability of LLM-assisted, interactive glossing workflows that are more controllable and interpretable than fully automatic, task-specific models, while highlighting current limitations and avenues for future improvements in instruction-following and morphology-aware prompting.

Abstract

Partly automated creation of interlinear glossed text (IGT) has the potential to assist in linguistic documentation. We argue that LLMs can make this process more accessible to linguists because of their capacity to follow natural-language instructions. We investigate the effectiveness of a retrieval-based LLM prompting approach to glossing, applied to the seven languages from the SIGMORPHON 2023 shared task. Our system beats the BERT-based shared task baseline for every language in the morpheme-level score category, and we show that a simple 3-best oracle has higher word-level scores than the challenge winner (a tuned sequence model) in five languages. In a case study on Tsez, we ask the LLM to automatically create and follow linguistic instructions, reducing errors on a confusing grammatical feature. Our results thus demonstrate the potential contributions which LLMs can make in interactive systems for glossing, both in making suggestions to human annotators and following directions.

Prompt and circumstance: A word-by-word LLM prompting approach to interlinear glossing for low-resource languages

TL;DR

The paper tackles the challenge of efficiently producing interlinear glossed text for low-resource languages by leveraging a retrieval-based, word-by-word prompting approach with a large language model. Using GPT-4o on seven SIGMORPHON 2023 languages, the method beats the Track 1 baseline at the morpheme level and, in many cases, the 3-best oracle matches or exceeds the challenge winner on word-level scores, with a Tsez case study showing that generated linguistic instructions can reduce error rates. The authors further demonstrate that disambiguation instructions for syncretic forms can meaningfully improve gloss accuracy in Tsez, illustrating the potential for human-in-the-loop guidance and linguistically informed prompts. Overall, the work supports the viability of LLM-assisted, interactive glossing workflows that are more controllable and interpretable than fully automatic, task-specific models, while highlighting current limitations and avenues for future improvements in instruction-following and morphology-aware prompting.

Abstract

Partly automated creation of interlinear glossed text (IGT) has the potential to assist in linguistic documentation. We argue that LLMs can make this process more accessible to linguists because of their capacity to follow natural-language instructions. We investigate the effectiveness of a retrieval-based LLM prompting approach to glossing, applied to the seven languages from the SIGMORPHON 2023 shared task. Our system beats the BERT-based shared task baseline for every language in the morpheme-level score category, and we show that a simple 3-best oracle has higher word-level scores than the challenge winner (a tuned sequence model) in five languages. In a case study on Tsez, we ask the LLM to automatically create and follow linguistic instructions, reducing errors on a confusing grammatical feature. Our results thus demonstrate the potential contributions which LLMs can make in interactive systems for glossing, both in making suggestions to human annotators and following directions.

Paper Structure

This paper contains 19 sections, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Automated glossing
  4. LLMs for low-resource NLP
  5. Task and data
  6. Prompt-based glossing
  7. Prompting
  8. Results and discussion
  9. Three-best oracle
  10. Disambiguation of syncretic forms
  11. Prompting
  12. Results and discussion
  13. Conclusion
  14. Canary testing for data leakage
  15. Morpheme-level one-best scores
  16. ...and 4 more sections

Figures (1)

  • Figure 1: Outline of Section \ref{['sec:disambig']}, showing the pipeline of instruction generation and inference-time disambiguation for a syncretic pair. Purple panels show LLM-generated text.