Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SMOL: Professionally translated parallel data for 115 under-represented languages

Isaac Caswell, Elizabeth Nielsen, Jiaming Luo, Colin Cherry, Geza Kovacs, Hadar Shemtov, Partha Talukdar, Dinesh Tewari, Baba Mamadi Diane, Djibrila Diane, Solo Farabado Cissé, Koulako Moussa Doumbouya, Edoardo Ferrante, Alessandro Guasoni, Christopher Homan, Mamadou K. Keita, Sudhamoy DebBarma, Ali Kuzhuget, David Anugraha, Muhammad Ravi Shulthan Habibi, Genta Indra Winata, Anthony Munthali, Sina Ahmadi, Andrei Chemyshev, Mingfei Lau, Jonathan Eng

TL;DR

This work introduces SMOL, a professionally translated parallel data suite aimed at enabling machine translation for 124 low-resource languages through two complementary datasets, SmolSent (sentence-level, token-coverage focused) and SmolDoc (document-level, topic-diverse). It details English-centered data selection, including a token-set-cover approach and a prompt-based document generation method, augmented by human-in-the-loop curation and factuality annotations. Finetuning and prompting experiments with Gemini 2.0 demonstrate substantial ChrF improvements across language pairs, with the strongest gains when combining SmolSent, SmolDoc, and Gatitos for diverse, non-English-centric languages. The dataset is openly released to support researchers and practitioners, though limitations such as potential English-source bias and multi-way data overfitting are acknowledged, guiding future work toward broader language coverage and robust evaluation.

Abstract

We open-source SMOL (Set of Maximal Overall Leverage), a suite of training data to unlock machine translation for low-resource languages. SMOL has been translated into 124 (and growing) under-resourced languages (125 language pairs), including many for which there exist no previous public resources, for a total of 6.1M translated tokens. SMOL comprises two sub-datasets, each carefully chosen for maximum impact given its size: SMOLSENT, a set of sentences chosen for broad unique token coverage, and SMOLDOC, a document-level resource focusing on a broad topic coverage. They join the already released GATITOS for a trifecta of paragraph, sentence, and token-level content. We demonstrate that using SMOL to prompt or fine-tune Large Language Models yields robust chrF improvements. In addition to translation, we provide factuality ratings and rationales for all documents in SMOLDOC, yielding the first factuality datasets for most of these languages.

SMOL: Professionally translated parallel data for 115 under-represented languages

TL;DR

This work introduces SMOL, a professionally translated parallel data suite aimed at enabling machine translation for 124 low-resource languages through two complementary datasets, SmolSent (sentence-level, token-coverage focused) and SmolDoc (document-level, topic-diverse). It details English-centered data selection, including a token-set-cover approach and a prompt-based document generation method, augmented by human-in-the-loop curation and factuality annotations. Finetuning and prompting experiments with Gemini 2.0 demonstrate substantial ChrF improvements across language pairs, with the strongest gains when combining SmolSent, SmolDoc, and Gatitos for diverse, non-English-centric languages. The dataset is openly released to support researchers and practitioners, though limitations such as potential English-source bias and multi-way data overfitting are acknowledged, guiding future work toward broader language coverage and robust evaluation.

Abstract

We open-source SMOL (Set of Maximal Overall Leverage), a suite of training data to unlock machine translation for low-resource languages. SMOL has been translated into 124 (and growing) under-resourced languages (125 language pairs), including many for which there exist no previous public resources, for a total of 6.1M translated tokens. SMOL comprises two sub-datasets, each carefully chosen for maximum impact given its size: SMOLSENT, a set of sentences chosen for broad unique token coverage, and SMOLDOC, a document-level resource focusing on a broad topic coverage. They join the already released GATITOS for a trifecta of paragraph, sentence, and token-level content. We demonstrate that using SMOL to prompt or fine-tune Large Language Models yields robust chrF improvements. In addition to translation, we provide factuality ratings and rationales for all documents in SMOLDOC, yielding the first factuality datasets for most of these languages.

Paper Structure

This paper contains 49 sections, 2 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Text Selection
  4. SmolSent: Token Set Cover
  5. Preliminary work on Token Set-Cover
  6. Researcher in the Loop (RITL)
  7. SmolDoc: LLMs with prompt mesh
  8. Corpus Diversity Ranking for SmolDoc
  9. Language Tiers for SmolDoc
  10. Non-English-centric translations
  11. Data Collection and Verification
  12. Finetuning and In-Context Learning
  13. Evaluation
  14. Finetuning Setup and Results
  15. The Problem with xx$\rightarrow$en training
  16. ...and 34 more sections

Figures (2)

  • Figure 1: Training curves (ChrF) for finetuned models on a subset of 37 en$\rightarrow$xx language pairs, trained on SmolDoc, SmolSent, and their combination Both.
  • Figure 2: SmolDoc factuality ratings.