Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Introducing cosmosGPT: Monolingual Training for Turkish Language Models

H. Toprak Kesgin, M. Kaan Yuce, Eren Dogan, M. Egemen Uzun, Atahan Uz, H. Emre Seyrek, Ahmed Zeer, M. Fatih Amasyali

TL;DR

This work addresses the challenge of developing effective Turkish language models with limited monolingual data by training cosmosGPT Medium and Large exclusively on Turkish corpora. It introduces new finetuning and evaluation datasets, and performs a comprehensive, human-assisted and automatic evaluation against Turkish and multilingual models, showing that monolingual CosmosGPT variants can match or approach the capabilities of much larger multilingual systems. Key contributions include a Turkish-only training corpus of 275GB, a 50,257-token tokenizer, instruction-following fine-tuning datasets (MH, BM, and variants), and a robust evaluation protocol combining ROUGE, cosine similarity, and ELO-based human judgments. The findings highlight the value of language-specific data curation and targeted instruction-following fine-tuning, with implications for cost-efficient, high-quality NLP in low-resource languages. The authors also outline future directions such as Direct Preference Optimization and reinforcement learning to further enhance performance and adaptability.

Abstract

The number of open source language models that can produce Turkish is increasing day by day, as in other languages. In order to create the basic versions of such models, the training of multilingual models is usually continued with Turkish corpora. The alternative is to train the model with only Turkish corpora. In this study, we first introduce the cosmosGPT models that we created with this alternative method. Then, we introduce new finetune datasets for basic language models to fulfill user requests and new evaluation datasets for measuring the capabilities of Turkish language models. Finally, a comprehensive comparison of the adapted Turkish language models on different capabilities is presented. The results show that the language models we built with the monolingual corpus have promising performance despite being about 10 times smaller than the others.

Introducing cosmosGPT: Monolingual Training for Turkish Language Models

TL;DR

This work addresses the challenge of developing effective Turkish language models with limited monolingual data by training cosmosGPT Medium and Large exclusively on Turkish corpora. It introduces new finetuning and evaluation datasets, and performs a comprehensive, human-assisted and automatic evaluation against Turkish and multilingual models, showing that monolingual CosmosGPT variants can match or approach the capabilities of much larger multilingual systems. Key contributions include a Turkish-only training corpus of 275GB, a 50,257-token tokenizer, instruction-following fine-tuning datasets (MH, BM, and variants), and a robust evaluation protocol combining ROUGE, cosine similarity, and ELO-based human judgments. The findings highlight the value of language-specific data curation and targeted instruction-following fine-tuning, with implications for cost-efficient, high-quality NLP in low-resource languages. The authors also outline future directions such as Direct Preference Optimization and reinforcement learning to further enhance performance and adaptability.

Abstract

The number of open source language models that can produce Turkish is increasing day by day, as in other languages. In order to create the basic versions of such models, the training of multilingual models is usually continued with Turkish corpora. The alternative is to train the model with only Turkish corpora. In this study, we first introduce the cosmosGPT models that we created with this alternative method. Then, we introduce new finetune datasets for basic language models to fulfill user requests and new evaluation datasets for measuring the capabilities of Turkish language models. Finally, a comprehensive comparison of the adapted Turkish language models on different capabilities is presented. The results show that the language models we built with the monolingual corpus have promising performance despite being about 10 times smaller than the others.
Paper Structure (14 sections, 3 figures, 7 tables)

This paper contains 14 sections, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Training GPT models in Turkish
  3. Creating Finetune Dataset
  4. Filtering & Combination
  5. Creating Evaluation Datasets
  6. Evaluation Criteria
  7. Comparison of Fine-tune Datasets Over GPT-2 Large Models
  8. Score Analysis
  9. Comparison of Turkish Language Models
  10. Scores of Models According to Evaluation Metrics
  11. Categorical Task Based Comparison
  12. Analysis of Comparison Results
  13. Inter-Metric Correlation Analysis
  14. Conclusion and Future Work

Figures (3)

  • Figure 1: ELO Ratings According to The 'V' Dataset
  • Figure 2: Scores According to Categorical Tasks
  • Figure 3: Metric Correlations