Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Kakugo: Distillation of Low-Resource Languages into Small Language Models

Peter Devine, Mardhiyah Sanni, Farid Adilazuarda, Julieta Gil Loizaga, Barry Haddow

TL;DR

Kakugo addresses the gap in accessible, generalist SLMs for low-resource languages by introducing a fully automated pipeline that distills the capabilities of a large teacher LLM into small monolingual models. It combines three data-generation streams (topic, scenario, and context prompts with reasoning traces) and translated data, then fine-tunes a compact student model within a cost-effective framework (<$50 per language). The approach yields consistent improvements across translation, classification, and question answering benchmarks for 54 languages, and manual evaluations favor Kakugo over the baseline. This work provides a scalable, community-friendly method to deploy AI in low-resource languages while emphasizing data diversity, affordability, and practical deployment considerations.

Abstract

We present Kakugo, a novel and cost-effective pipeline designed to train general-purpose Small Language Models (SLMs) for low-resource languages using only the language name as input. By using a large teacher model to generate synthetic prompts and translate instruction datasets, we produced training data and SLMs for 54 low-resource languages. Evaluations across a diverse set of general natural language processing tasks, including translation, classification, and question answering, demonstrate that our pipeline consistently improves performance over base models. With a total generation and training cost of under $50 per language, Kakugo offers an accessible method for communities to develop language-specific AI.

Kakugo: Distillation of Low-Resource Languages into Small Language Models

TL;DR

Kakugo addresses the gap in accessible, generalist SLMs for low-resource languages by introducing a fully automated pipeline that distills the capabilities of a large teacher LLM into small monolingual models. It combines three data-generation streams (topic, scenario, and context prompts with reasoning traces) and translated data, then fine-tunes a compact student model within a cost-effective framework (<$50 per language). The approach yields consistent improvements across translation, classification, and question answering benchmarks for 54 languages, and manual evaluations favor Kakugo over the baseline. This work provides a scalable, community-friendly method to deploy AI in low-resource languages while emphasizing data diversity, affordability, and practical deployment considerations.

Abstract

We present Kakugo, a novel and cost-effective pipeline designed to train general-purpose Small Language Models (SLMs) for low-resource languages using only the language name as input. By using a large teacher model to generate synthetic prompts and translate instruction datasets, we produced training data and SLMs for 54 low-resource languages. Evaluations across a diverse set of general natural language processing tasks, including translation, classification, and question answering, demonstrate that our pipeline consistently improves performance over base models. With a total generation and training cost of under $50 per language, Kakugo offers an accessible method for communities to develop language-specific AI.
Paper Structure (34 sections, 2 figures, 6 tables)

This paper contains 34 sections, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Kakugo model generation pipeline
  4. Data creation
  5. Generating synthetic prompts
  6. Topic-based prompt generation
  7. Scenario-based prompt generation
  8. Context-based prompt generation
  9. Generating responses from prompts
  10. Translating data
  11. Model training
  12. Experiments
  13. Language selection
  14. Data subsets
  15. Evaluation benchmarks
  16. ...and 19 more sections

Figures (2)

  • Figure 1: The Kakugo pipeline. Given a language name (e.g., "Javanese"), a teacher model generates synthetic conversational and instruction data which are used to train an SLM in that language.
  • Figure 2: Response preferences from native speakers between responses from our Kakugo model and the base model, Granite 4 Micro.