Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Boosting the Capabilities of Compact Models in Low-Data Contexts with Large Language Models and Retrieval-Augmented Generation

Bhargav Shandilya, Alexis Palmer

TL;DR

This paper proposes a retrieval augmented generation framework backed by a large language model (LLM) to correct the output of a smaller model for the linguistic task of morphological glossing, and shows that a compact, RAG-supported model is highly effective in data-scarce settings.

Abstract

The data and compute requirements of current language modeling technology pose challenges for the processing and analysis of low-resource languages. Declarative linguistic knowledge has the potential to partially bridge this data scarcity gap by providing models with useful inductive bias in the form of language-specific rules. In this paper, we propose a retrieval augmented generation (RAG) framework backed by a large language model (LLM) to correct the output of a smaller model for the linguistic task of morphological glossing. We leverage linguistic information to make up for the lack of data and trainable parameters, while allowing for inputs from written descriptive grammars interpreted and distilled through an LLM. The results demonstrate that significant leaps in performance and efficiency are possible with the right combination of: a) linguistic inputs in the form of grammars, b) the interpretive power of LLMs, and c) the trainability of smaller token classification networks. We show that a compact, RAG-supported model is highly effective in data-scarce settings, achieving a new state-of-the-art for this task and our target languages. Our work also offers documentary linguists a more reliable and more usable tool for morphological glossing by providing well-reasoned explanations and confidence scores for each output.

Boosting the Capabilities of Compact Models in Low-Data Contexts with Large Language Models and Retrieval-Augmented Generation

TL;DR

This paper proposes a retrieval augmented generation framework backed by a large language model (LLM) to correct the output of a smaller model for the linguistic task of morphological glossing, and shows that a compact, RAG-supported model is highly effective in data-scarce settings.

Abstract

The data and compute requirements of current language modeling technology pose challenges for the processing and analysis of low-resource languages. Declarative linguistic knowledge has the potential to partially bridge this data scarcity gap by providing models with useful inductive bias in the form of language-specific rules. In this paper, we propose a retrieval augmented generation (RAG) framework backed by a large language model (LLM) to correct the output of a smaller model for the linguistic task of morphological glossing. We leverage linguistic information to make up for the lack of data and trainable parameters, while allowing for inputs from written descriptive grammars interpreted and distilled through an LLM. The results demonstrate that significant leaps in performance and efficiency are possible with the right combination of: a) linguistic inputs in the form of grammars, b) the interpretive power of LLMs, and c) the trainability of smaller token classification networks. We show that a compact, RAG-supported model is highly effective in data-scarce settings, achieving a new state-of-the-art for this task and our target languages. Our work also offers documentary linguists a more reliable and more usable tool for morphological glossing by providing well-reasoned explanations and confidence scores for each output.
Paper Structure (24 sections, 10 equations, 1 figure, 6 tables)

This paper contains 24 sections, 10 equations, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. The glossing task
  4. Related Work
  5. Methodology
  6. Using Naive RAG to correct the output of a smaller model
  7. Generating Labeling Justifications and Confidence Scores
  8. Modular RAG: Training the retriever with the sequence to sequence model
  9. Baseline Glossing Models
  10. Uspanteko and Arapaho: Data and Grammars
  11. Data.
  12. Experiments and results
  13. Qualitative analysis: usability
  14. Glossing error types.
  15. Quality of explanations.
  16. ...and 9 more sections

Figures (1)

  • Figure 1: One Uspanteko sentence, with its original gloss, a predicted gloss, and an explanation from our IGT-RAG model. For each morpheme in the sentence, the model describes which section of the provided Uspanteko grammar it used to make its labeling decision.