Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Instruction Tuning on Public Government and Cultural Data for Low-Resource Language: a Case Study in Kazakh

Nurkhan Laiyk, Daniil Orel, Rituraj Joshi, Maiya Goloburda, Yuxia Wang, Preslav Nakov, Fajri Koto

Abstract

Instruction tuning in low-resource languages remains underexplored due to limited text data, particularly in government and cultural domains. To address this, we introduce and open-source a large-scale (10,600 samples) instruction-following (IFT) dataset, covering key institutional and cultural knowledge relevant to Kazakhstan. Our dataset enhances LLMs' understanding of procedural, legal, and structural governance topics. We employ LLM-assisted data generation, comparing open-weight and closed-weight models for dataset construction, and select GPT-4o as the backbone. Each entity of our dataset undergoes full manual verification to ensure high quality. We also show that fine-tuning Qwen, Falcon, and Gemma on our dataset leads to consistent performance improvements in both multiple-choice and generative tasks, demonstrating the potential of LLM-assisted instruction tuning for low-resource languages.

Instruction Tuning on Public Government and Cultural Data for Low-Resource Language: a Case Study in Kazakh

Abstract

Instruction tuning in low-resource languages remains underexplored due to limited text data, particularly in government and cultural domains. To address this, we introduce and open-source a large-scale (10,600 samples) instruction-following (IFT) dataset, covering key institutional and cultural knowledge relevant to Kazakhstan. Our dataset enhances LLMs' understanding of procedural, legal, and structural governance topics. We employ LLM-assisted data generation, comparing open-weight and closed-weight models for dataset construction, and select GPT-4o as the backbone. Each entity of our dataset undergoes full manual verification to ensure high quality. We also show that fine-tuning Qwen, Falcon, and Gemma on our dataset leads to consistent performance improvements in both multiple-choice and generative tasks, demonstrating the potential of LLM-assisted instruction tuning for low-resource languages.

Paper Structure

This paper contains 53 sections, 18 figures, 20 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Instruction-Tuning Datasets in English
  4. Instruction Tuning Datasets for Medium- to Low-Resource Languages
  5. Existing datasets in Kazakh
  6. Background
  7. Kazakhstan's Institutional Structure and Public Governance.
  8. Data
  9. Document Source
  10. GovSet
  11. CultSet
  12. LLM-assisted Data Generation
  13. Human Evaluation Across LLMs
  14. Manual Post-Editing
  15. Final Data Overview
  16. ...and 38 more sections

Figures (18)

  • Figure 1: Overview of the end-to-end process for constructing GovSet and CultSet datasets. English translations are for illustration purposes.
  • Figure 4: Conversational data preference evaluation.
  • Figure 7: Inner annotator agreement across annotators for correctness, completeness, and fluency, measured using Pearson correlation.
  • Figure 8: Inner-annotator agreement for generation evaluation, measured using Cohen’s Kappa.
  • Figure : (a) CultSet
  • ...and 13 more figures