Optimizing Large Language Models for Turkish: New Methodologies in Corpus Selection and Training

TL;DR

This work tackles Turkish data scarcity in multilingual large language models by translating English corpora and integrating LLM-generated Turkish datasets, coupled with optimized corpus selection and a linear model merging strategy. The authors train 7–8B parameter Turkish language variants based on Llama3, apply targeted data selection (SKWO, Stories, OpenOrca), and merge models to yield notable performance gains, validated through both automated benchmarks (ARC, HellaSwag, GSM8K, MMLU, TruthfulQA, Winogrande) and human judge voting using ELO scores. Key contributions include new Turkish adaptation datasets, refined cross-lingual corpus construction, and evidence that small-model optimizations translate to larger models, with synthetic data playing a crucial role for low-resource languages. The results have practical implications for developing robust Turkish language technologies within multilingual ecosystems, supported by cloud TPU resources and Turkish funding agencies.

Abstract

In this study, we develop and assess new corpus selection and training methodologies to improve the effectiveness of Turkish language models. Specifically, we adapted Large Language Model generated datasets and translated English datasets into Turkish, integrating these resources into the training process. This approach led to substantial enhancements in model accuracy for both few-shot and zero-shot learning scenarios. Furthermore, the merging of these adapted models was found to markedly improve their performance. Human evaluative metrics, including task-specific performance assessments, further demonstrated that these adapted models possess a greater aptitude for comprehending the Turkish language and addressing logic-based queries. This research underscores the importance of refining corpus selection strategies to optimize the performance of multilingual models, particularly for under-resourced languages like Turkish.

Figures (4)

• Figure 1: Models Performance Across Categories
• Figure 2: Human Judges Preferences Correlation Matrix
• Figure 3: Metric Correlation Matrix
• Figure 4: Correlation between categories