Setting up the Data Printer with Improved English to Ukrainian Machine Translation

TL;DR

The paper addresses the problem of building high-quality English–Ukrainian translation to accelerate Ukrainian data curation for large language models. It proposes a two-phase supervised finetuning approach on a decoder-only Transformer (Dragoman) using a noisy Paracrawl set and a high-quality Extended Multi30K subset, with adapters trained via $q\text{LoRA}$ and a conditional likelihood objective $p_{\theta,\phi}(Y|X)$. Key contributions include effective heuristic filtering of Paracrawl, perplexity-based unsupervised data selection on Extended Multi30K, and demonstrated BLEU gains on FLORES devtest, with competitive performance relative to state-of-the-art encoder–decoder models. The results imply practical pathways to improve translation quality for Ukrainian in the context of LLM development, while also highlighting limitations of single-sentence training, tokenizer inefficiencies, and evaluation metrics. Overall, the work provides a replicable data-engineering recipe to boost Ukrainian translation capabilities for large-scale multilingual models.

Abstract

To build large language models for Ukrainian we need to expand our corpora with large amounts of new algorithmic tasks expressed in natural language. Examples of task performance expressed in English are abundant, so with a high-quality translation system our community will be enabled to curate datasets faster. To aid this goal, we introduce a recipe to build a translation system using supervised finetuning of a large pretrained language model with a noisy parallel dataset of 3M pairs of Ukrainian and English sentences followed by a second phase of training using 17K examples selected by k-fold perplexity filtering on another dataset of higher quality. Our decoder-only model named Dragoman beats performance of previous state of the art encoder-decoder models on the FLORES devtest set.

Figures (4)

• Figure 1: Distributions of sentence log probabilities for each fold superimposed on top of each other. Every bar color represents a unique fold; every vertical line denotes a 60th percentile cutoff threshold. The best percentile is chosen using grid search shown in \ref{['tab:pplthresholds']}.
• Figure 2: Basic 2-shot prompt used for few-shot translation. [INST] prefixes the beginning of the source sentence and [/INST] denotes the beginning of the target translation. These separators are chosen arbitrarily (as in finetuning) and are not special vocabulary items, even though they bear visual resemblance to them.
• Figure 3: Few-shot translation with contextual prompting allows the model to learn named entities on the fly. Without context, the model makes a wrong guess trying to transliterate the abbreviation.
• Figure 4: Comparison of tokenizer compression rates between English and Ukrainian using the Mistral-7B tokenizer on the FLORES dev set.