Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Bytes to Borsch: Fine-Tuning Gemma and Mistral for the Ukrainian Language Representation

Artur Kiulian, Anton Polishko, Mykola Khandoga, Oryna Chubych, Jack Connor, Raghav Ravishankar, Adarsh Shirawalmath

TL;DR

Fine-tuning open-source Gemma and Mistral LLMs with Ukrainian data addresses language bias and improves Ukrainian capabilities. The authors introduce UKID, a Ukrainian-context knowledge/instruction dataset, and ULIB benchmarks to guide future fine-tuning. Through LoRA-based fine-tuning on Gemma variants and Mistral using Ukrainian corpora (UAlpaca, ZNO, UKID) the work shows notable gains for Ukrainian tasks, though proprietary models still outperform large baselines. The study highlights emergent code-switching (Azirivka) and argues for broader investments in language-specific AI to preserve linguistic diversity and enable inclusive digital infrastructure.

Abstract

In the rapidly advancing field of AI and NLP, generative large language models (LLMs) stand at the forefront of innovation, showcasing unparalleled abilities in text understanding and generation. However, the limited representation of low-resource languages like Ukrainian poses a notable challenge, restricting the reach and relevance of this technology. Our paper addresses this by fine-tuning the open-source Gemma and Mistral LLMs with Ukrainian datasets, aiming to improve their linguistic proficiency and benchmarking them against other existing models capable of processing Ukrainian language. This endeavor not only aims to mitigate language bias in technology but also promotes inclusivity in the digital realm. Our transparent and reproducible approach encourages further NLP research and development. Additionally, we present the Ukrainian Knowledge and Instruction Dataset (UKID) to aid future efforts in language model fine-tuning. Our research not only advances the field of NLP but also highlights the importance of linguistic diversity in AI, which is crucial for cultural preservation, education, and expanding AI's global utility. Ultimately, we advocate for a future where technology is inclusive, enabling AI to communicate effectively across all languages, especially those currently underrepresented.

From Bytes to Borsch: Fine-Tuning Gemma and Mistral for the Ukrainian Language Representation

TL;DR

Fine-tuning open-source Gemma and Mistral LLMs with Ukrainian data addresses language bias and improves Ukrainian capabilities. The authors introduce UKID, a Ukrainian-context knowledge/instruction dataset, and ULIB benchmarks to guide future fine-tuning. Through LoRA-based fine-tuning on Gemma variants and Mistral using Ukrainian corpora (UAlpaca, ZNO, UKID) the work shows notable gains for Ukrainian tasks, though proprietary models still outperform large baselines. The study highlights emergent code-switching (Azirivka) and argues for broader investments in language-specific AI to preserve linguistic diversity and enable inclusive digital infrastructure.

Abstract

In the rapidly advancing field of AI and NLP, generative large language models (LLMs) stand at the forefront of innovation, showcasing unparalleled abilities in text understanding and generation. However, the limited representation of low-resource languages like Ukrainian poses a notable challenge, restricting the reach and relevance of this technology. Our paper addresses this by fine-tuning the open-source Gemma and Mistral LLMs with Ukrainian datasets, aiming to improve their linguistic proficiency and benchmarking them against other existing models capable of processing Ukrainian language. This endeavor not only aims to mitigate language bias in technology but also promotes inclusivity in the digital realm. Our transparent and reproducible approach encourages further NLP research and development. Additionally, we present the Ukrainian Knowledge and Instruction Dataset (UKID) to aid future efforts in language model fine-tuning. Our research not only advances the field of NLP but also highlights the importance of linguistic diversity in AI, which is crucial for cultural preservation, education, and expanding AI's global utility. Ultimately, we advocate for a future where technology is inclusive, enabling AI to communicate effectively across all languages, especially those currently underrepresented.
Paper Structure (24 sections, 2 figures, 3 tables)

This paper contains 24 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. The transformer era
  4. Motivation and objective
  5. Ukrainian sector of the LLMs
  6. Objectives
  7. Dataset and the experimental setup
  8. Dataset collection
  9. UKID methodology and construction
  10. Fine-tuning
  11. Gemma models
  12. Mistral model
  13. Benchmarking results
  14. Multiple choice questions
  15. Open questions
  16. ...and 9 more sections

Figures (2)

  • Figure 1: Example of erroneous model inference in Ukrainian.
  • Figure 2: Prompt to generate UKID examples