Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hire a Linguist!: Learning Endangered Languages with In-Context Linguistic Descriptions

Kexun Zhang, Yee Man Choi, Zhenqiao Song, Taiqi He, William Yang Wang, Lei Li

TL;DR

This work proposes LINGOLLM, a training-free approach to enable an LLM to process unseen languages that hardly occur in its pre-training, and elevates translation capability from GPT-4's 0 to 10.5 BLEU for 10 language directions.

Abstract

How can large language models (LLMs) process and translate endangered languages? Many languages lack a large corpus to train a decent LLM; therefore existing LLMs rarely perform well in unseen, endangered languages. On the contrary, we observe that 2000 endangered languages, though without a large corpus, have a grammar book or a dictionary. We propose LINGOLLM, a training-free approach to enable an LLM to process unseen languages that hardly occur in its pre-training. Our key insight is to demonstrate linguistic knowledge of an unseen language in an LLM's prompt, including a dictionary, a grammar book, and morphologically analyzed input text. We implement LINGOLLM on top of two models, GPT-4 and Mixtral, and evaluate their performance on 5 tasks across 8 endangered or low-resource languages. Our results show that LINGOLLM elevates translation capability from GPT-4's 0 to 10.5 BLEU for 10 language directions. Our findings demonstrate the tremendous value of linguistic knowledge in the age of LLMs for endangered languages. Our data, code, and model generations can be found at https://github.com/LLiLab/llm4endangeredlang.

Hire a Linguist!: Learning Endangered Languages with In-Context Linguistic Descriptions

TL;DR

This work proposes LINGOLLM, a training-free approach to enable an LLM to process unseen languages that hardly occur in its pre-training, and elevates translation capability from GPT-4's 0 to 10.5 BLEU for 10 language directions.

Abstract

How can large language models (LLMs) process and translate endangered languages? Many languages lack a large corpus to train a decent LLM; therefore existing LLMs rarely perform well in unseen, endangered languages. On the contrary, we observe that 2000 endangered languages, though without a large corpus, have a grammar book or a dictionary. We propose LINGOLLM, a training-free approach to enable an LLM to process unseen languages that hardly occur in its pre-training. Our key insight is to demonstrate linguistic knowledge of an unseen language in an LLM's prompt, including a dictionary, a grammar book, and morphologically analyzed input text. We implement LINGOLLM on top of two models, GPT-4 and Mixtral, and evaluate their performance on 5 tasks across 8 endangered or low-resource languages. Our results show that LINGOLLM elevates translation capability from GPT-4's 0 to 10.5 BLEU for 10 language directions. Our findings demonstrate the tremendous value of linguistic knowledge in the age of LLMs for endangered languages. Our data, code, and model generations can be found at https://github.com/LLiLab/llm4endangeredlang.
Paper Structure (30 sections, 5 figures, 8 tables)

This paper contains 30 sections, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The LingoLLM Approach
  4. Morphological Analysis: Source Sentence $\rightarrow$ Morphemes
  5. Dictionary Mapping: Morphemes $\rightarrow$ Gloss
  6. Incorporating Grammar Knowledge Gloss $\rightarrow$ Translation and Beyond
  7. Experiment
  8. Experiment Setup
  9. Baselines
  10. Benchmarks and Metrics
  11. Results
  12. Ablation and Analysis
  13. Morphological analysis helps.
  14. High-quality dictionary helps.
  15. How to make use of grammar knowledge.
  16. ...and 15 more sections

Figures (5)

  • Figure 1: Among the world's $\sim$7000 languages, 95% don't have enough data (>100K sentences) for training LLMs bapna2022building, while most have a grammar book (60%) or dictionary (75%) nordhoff2011glottolog, including many endangered languages moseley2010atlas. Therefore, we utilize these linguistic descriptions to bring LLMs to endangered languages.
  • Figure 2: LingoLLM significantly outperforms GPT-4 on 5 NLP tasks across 8 endangered or low-resource languages.
  • Figure 3: LingoLLM uses a morphological analyzer to transform the source sentence into morphemes, looks up the morphemes in a dictionary to obtain the gloss, and finally feeds both the gloss and a grammar book to an LLM to obtain the result.
  • Figure 4: LingoLLM significantly improves LLMs' ability to select correct responses. On all three endangered languages, it achieves a performance comparable to high-resource language inputs.
  • Figure 5: When more dictionary entries are masked out, LingoLLM's performance drops. When other relevant words referenced in the dictionary are not considered, LingoLLM's performance also drops.