Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can LLMs Really Learn to Translate a Low-Resource Language from One Grammar Book?

Seth Aycock, David Stap, Di Wu, Christof Monz, Khalil Sima'an

TL;DR

This study scrutinizes whether LLMs can learn to translate extremely low-resource languages from a grammar book, focusing on Kalamang and extending to Nepali and Guarani. It disentangles parallel sentence data from grammatical explanations, finding that translation performance is primarily driven by parallel data within the book, with grammatical explanations offering negligible gains. Fine-tuning small encoder-decoder MT models on the same parallel data achieves near-parity with long-context LLM prompts, suggesting a cheaper, data-efficient path for XLR translation. However, the authors show that typological prompts can improve linguistics-oriented tasks like grammaticality judgment and IGT prediction, indicating grammar knowledge helps when framed for the right task. Overall, the work argues for task-appropriate data collection: parallel data for translation and typological/grammatical data for linguistic analysis, with minimal reliance on grammar explanations for MT.

Abstract

Extremely low-resource (XLR) languages lack substantial corpora for training NLP models, motivating the use of all available resources such as dictionaries and grammar books. Machine Translation from One Book (Tanzer et al., 2024) suggests that prompting long-context LLMs with one grammar book enables English-Kalamang translation, an XLR language unseen by LLMs - a noteworthy case of linguistics helping an NLP task. We investigate the source of this translation ability, finding almost all improvements stem from the book's parallel examples rather than its grammatical explanations. We find similar results for Nepali and Guarani, seen low-resource languages, and we achieve performance comparable to an LLM with a grammar book by simply fine-tuning an encoder-decoder translation model. We then investigate where grammar books help by testing two linguistic tasks, grammaticality judgment and gloss prediction, and we explore what kind of grammatical knowledge helps by introducing a typological feature prompt that achieves leading results on these more relevant tasks. We thus emphasise the importance of task-appropriate data for XLR languages: parallel examples for translation, and grammatical data for linguistic tasks. As we find no evidence that long-context LLMs can make effective use of grammatical explanations for XLR translation, we conclude data collection for multilingual XLR tasks such as translation is best focused on parallel data over linguistic description.

Can LLMs Really Learn to Translate a Low-Resource Language from One Grammar Book?

TL;DR

This study scrutinizes whether LLMs can learn to translate extremely low-resource languages from a grammar book, focusing on Kalamang and extending to Nepali and Guarani. It disentangles parallel sentence data from grammatical explanations, finding that translation performance is primarily driven by parallel data within the book, with grammatical explanations offering negligible gains. Fine-tuning small encoder-decoder MT models on the same parallel data achieves near-parity with long-context LLM prompts, suggesting a cheaper, data-efficient path for XLR translation. However, the authors show that typological prompts can improve linguistics-oriented tasks like grammaticality judgment and IGT prediction, indicating grammar knowledge helps when framed for the right task. Overall, the work argues for task-appropriate data collection: parallel data for translation and typological/grammatical data for linguistic analysis, with minimal reliance on grammar explanations for MT.

Abstract

Extremely low-resource (XLR) languages lack substantial corpora for training NLP models, motivating the use of all available resources such as dictionaries and grammar books. Machine Translation from One Book (Tanzer et al., 2024) suggests that prompting long-context LLMs with one grammar book enables English-Kalamang translation, an XLR language unseen by LLMs - a noteworthy case of linguistics helping an NLP task. We investigate the source of this translation ability, finding almost all improvements stem from the book's parallel examples rather than its grammatical explanations. We find similar results for Nepali and Guarani, seen low-resource languages, and we achieve performance comparable to an LLM with a grammar book by simply fine-tuning an encoder-decoder translation model. We then investigate where grammar books help by testing two linguistic tasks, grammaticality judgment and gloss prediction, and we explore what kind of grammatical knowledge helps by introducing a typological feature prompt that achieves leading results on these more relevant tasks. We thus emphasise the importance of task-appropriate data for XLR languages: parallel examples for translation, and grammatical data for linguistic tasks. As we find no evidence that long-context LLMs can make effective use of grammatical explanations for XLR translation, we conclude data collection for multilingual XLR tasks such as translation is best focused on parallel data over linguistic description.
Paper Structure (34 sections, 3 figures, 11 tables)

This paper contains 34 sections, 3 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Grammar for low-resource machine translation
  4. Linguistics in NLP
  5. Interlinear gloss prediction
  6. Methodology
  7. Grammar Books for Translation
  8. Neural Machine Translation Approaches
  9. Typological Feature Prompting
  10. Grammaticality Judgment
  11. Interlinear Glossed Text Prediction
  12. Experimental Setup
  13. Data
  14. Models
  15. Evaluation
  16. ...and 19 more sections

Figures (3)

  • Figure 1: Grammaticality judgment accuracy in kgv; for reference in eng tests, Gemini scores 100%, 99%, and 100% respectively. Our prompt Typ + $\textsc{Book}_{p}$ performs best overall suggesting grammar can help LLMs for linguistic tasks.
  • Figure 2: Regression models of ChrF++ score against test type coverage for eng--kgv and kgv--eng translation with Gemini. Prompt settings are labelled with abbreviations for clarity. The plots show that translation performance can be statistically modelled by test set vocabulary coverage.
  • Figure 3: A brief passage from Visser22-GrammarKalamang, showing the format of $\textsc{Book}_{\text{non-para}}$ (above) and $\textsc{Book}_{\text{para}}$ (examples 17 and 18) explaining a morphological feature of Kalamang.