Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages

Artur Kiulian, Anton Polishko, Mykola Khandoga, Yevhen Kostiuk, Guillermo Gabrielli, Łukasz Gagała, Fadi Zaraket, Qusai Abu Obaida, Hrishikesh Garud, Wendy Wing Yee Mak, Dmytro Chaplynskyi, Selma Belhadj Amor, Grigol Peradze

TL;DR

This work proposes a model-agnostic cost-effective approach to developing bilingual base large language models (LLMs) to support English and any target language that mitigates the disproportionate penalization of underrepresented languages, promoting fairness and minimizing adverse phenomena such as code-switching and broken grammar.

Abstract

In this paper, we propose a model-agnostic cost-effective approach to developing bilingual base large language models (LLMs) to support English and any target language. The method includes vocabulary expansion, initialization of new embeddings, model training and evaluation. We performed our experiments with three languages, each using a non-Latin script - Ukrainian, Arabic, and Georgian. Our approach demonstrates improved language performance while reducing computational costs. It mitigates the disproportionate penalization of underrepresented languages, promoting fairness and minimizing adverse phenomena such as code-switching and broken grammar. Additionally, we introduce new metrics to evaluate language quality, revealing that vocabulary size significantly impacts the quality of generated text.

From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages

TL;DR

This work proposes a model-agnostic cost-effective approach to developing bilingual base large language models (LLMs) to support English and any target language that mitigates the disproportionate penalization of underrepresented languages, promoting fairness and minimizing adverse phenomena such as code-switching and broken grammar.

Abstract

In this paper, we propose a model-agnostic cost-effective approach to developing bilingual base large language models (LLMs) to support English and any target language. The method includes vocabulary expansion, initialization of new embeddings, model training and evaluation. We performed our experiments with three languages, each using a non-Latin script - Ukrainian, Arabic, and Georgian. Our approach demonstrates improved language performance while reducing computational costs. It mitigates the disproportionate penalization of underrepresented languages, promoting fairness and minimizing adverse phenomena such as code-switching and broken grammar. Additionally, we introduce new metrics to evaluate language quality, revealing that vocabulary size significantly impacts the quality of generated text.

Paper Structure

This paper contains 33 sections, 1 equation, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Vocabulary Extension Methodology
  5. Embeddings Initialization
  6. Continual pre-training
  7. Datasets
  8. Vocabulary Extension Datasets
  9. Continual Pre-training Datasets
  10. Experimental Setup
  11. Evaluation Metrics
  12. Tokenizer Fertility
  13. Non-Existing Words Ratio (NEWR)
  14. Code Switching Word Ratio (CSWR)
  15. Grammar Correctness Score (GCS)
  16. ...and 18 more sections

Figures (3)

  • Figure 1: Tokens adoption by Mistral model.
  • Figure 2: Ukrainian evaluation graph per training step. The name includes the embedding initialization technique: mean, residual, and NACHOS.
  • Figure 3: Arabic evaluation graph per training step. The name includes the embedding initialization technique: FOCUS, NACHOS, and mean.