Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From MTEB to MTOB: Retrieval-Augmented Classification for Descriptive Grammars

Albert Kornilov, Tatiana Shavrina

TL;DR

The paper tackles the problem of scaling NLP and machine translation to under-resourced languages by extracting typological features from descriptive grammars using Retrieval-Augmented Generation (RAG). It introduces two benchmarks and an open-source RAG pipeline to evaluate how well large language models can read grammars and classify typological features, including WALS and Grambank attributes. Key findings show that traditional BM25 retrieval is competitive with modern embeddings for this domain, and that end-to-end RAG pipelines can outperform baselines, though linguistic descriptions still pose non-trivial challenges and chain-of-thought prompts yield mixed benefits. Overall, the work provides a practical framework and data resources to advance typological extraction and MT for low-resource languages, while highlighting the need for standardized grammars and further methodological improvements.

Abstract

Recent advances in language modeling have demonstrated significant improvements in zero-shot capabilities, including in-context learning, instruction following, and machine translation for extremely under-resourced languages (Tanzer et al., 2024). However, many languages with limited written resources rely primarily on formal descriptions of grammar and vocabulary. In this paper, we introduce a set of benchmarks to evaluate how well models can extract and classify information from the complex descriptions found in linguistic grammars. We present a Retrieval-Augmented Generation (RAG)-based approach that leverages these descriptions for downstream tasks such as machine translation. Our benchmarks encompass linguistic descriptions for 248 languages across 142 language families, focusing on typological features from WALS and Grambank. This set of benchmarks offers the first comprehensive evaluation of language models' in-context ability to accurately interpret and extract linguistic features, providing a critical resource for scaling NLP to low-resource languages. The code and data are publicly available at \url{https://github.com/al-the-eigenvalue/RAG-on-grammars}.

From MTEB to MTOB: Retrieval-Augmented Classification for Descriptive Grammars

TL;DR

The paper tackles the problem of scaling NLP and machine translation to under-resourced languages by extracting typological features from descriptive grammars using Retrieval-Augmented Generation (RAG). It introduces two benchmarks and an open-source RAG pipeline to evaluate how well large language models can read grammars and classify typological features, including WALS and Grambank attributes. Key findings show that traditional BM25 retrieval is competitive with modern embeddings for this domain, and that end-to-end RAG pipelines can outperform baselines, though linguistic descriptions still pose non-trivial challenges and chain-of-thought prompts yield mixed benefits. Overall, the work provides a practical framework and data resources to advance typological extraction and MT for low-resource languages, while highlighting the need for standardized grammars and further methodological improvements.

Abstract

Recent advances in language modeling have demonstrated significant improvements in zero-shot capabilities, including in-context learning, instruction following, and machine translation for extremely under-resourced languages (Tanzer et al., 2024). However, many languages with limited written resources rely primarily on formal descriptions of grammar and vocabulary. In this paper, we introduce a set of benchmarks to evaluate how well models can extract and classify information from the complex descriptions found in linguistic grammars. We present a Retrieval-Augmented Generation (RAG)-based approach that leverages these descriptions for downstream tasks such as machine translation. Our benchmarks encompass linguistic descriptions for 248 languages across 142 language families, focusing on typological features from WALS and Grambank. This set of benchmarks offers the first comprehensive evaluation of language models' in-context ability to accurately interpret and extract linguistic features, providing a critical resource for scaling NLP to low-resource languages. The code and data are publicly available at \url{https://github.com/al-the-eigenvalue/RAG-on-grammars}.

Paper Structure

This paper contains 29 sections, 8 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Retrieval-Augmented Generation
  4. Zero-shot Learning
  5. Grammar Use in NLP
  6. Extracting Typological Features from Grammars
  7. Method and Overall Architecture
  8. The Benchmark for Rerankers
  9. Data
  10. Results: Evaluating Rerankers
  11. The Benchmark for RAG
  12. Data
  13. Results: Evaluating the RAG Pipeline
  14. Discussion
  15. Conclusion
  16. ...and 14 more sections

Figures (8)

  • Figure 1: NDCG@k across all values of k for best performing configurations for each model.
  • Figure 2: Mean NDCG@k for all grammars: Wikipedia Summary vs Term Only.
  • Figure :
  • Figure :
  • Figure :
  • ...and 3 more figures