Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Racka: Efficient Hungarian LLM Adaptation on Academic Infrastructure

Zsolt Csibi, Bence György Gortka, Natabara Gyöngyössy, Kornél Nagy, Dávid Márk Nemeskey, Martin Sallai, András Simonyi, András Márk Szekeres, Gábor Palkó

TL;DR

Racka addresses the resource gap for Hungarian LLMs by presenting a lightweight 4B model that is continually pretrained via LoRA on a Qwen-3-4B backbone, trained on a 160B-token multilingual corpus using public national HPC infrastructure. It combines a Hungarian-adapted tokenizer with a carefully designed data mix to mitigate catastrophic forgetting while preserving high-resource language capabilities. The results show modesto yet stable improvements in tokenization efficiency and competitive performance on several Hungarian benchmarks relative to larger models, illustrating a practical path toward digital sovereignty and accessible LLM development. The work also emphasizes openness by planning to release the model and tokenizer for community use on publicly available infrastructure.

Abstract

We present Racka, a lightweight, continually pretrained large language model designed to bridge the resource gap between Hungarian and high-resource languages such as English and German. Racka employs parameter-efficient continual pretraining via Low-Rank Adaptation (LoRA) on a Qwen-3 4B backbone, making the recipe practical on A100 (40GB)-based HPC clusters with low inter-node bandwidth. To better match the training distribution, we replace and adapt the tokenizer, achieving substantially improved tokenization fertility for Hungarian while maintaining competitive performance in English and German. The model is trained on 160B subword tokens drawn from a mixture of internet and high-quality curated sources, with a composition of 44% Hungarian, 24% English, 21% German, and 11% code. This data mix is chosen to mitigate catastrophic forgetting and preserve high-resource language capabilities during continual pretraining. Our preliminary results indicate modest but stable results in language adaptation.

Racka: Efficient Hungarian LLM Adaptation on Academic Infrastructure

TL;DR

Racka addresses the resource gap for Hungarian LLMs by presenting a lightweight 4B model that is continually pretrained via LoRA on a Qwen-3-4B backbone, trained on a 160B-token multilingual corpus using public national HPC infrastructure. It combines a Hungarian-adapted tokenizer with a carefully designed data mix to mitigate catastrophic forgetting while preserving high-resource language capabilities. The results show modesto yet stable improvements in tokenization efficiency and competitive performance on several Hungarian benchmarks relative to larger models, illustrating a practical path toward digital sovereignty and accessible LLM development. The work also emphasizes openness by planning to release the model and tokenizer for community use on publicly available infrastructure.

Abstract

We present Racka, a lightweight, continually pretrained large language model designed to bridge the resource gap between Hungarian and high-resource languages such as English and German. Racka employs parameter-efficient continual pretraining via Low-Rank Adaptation (LoRA) on a Qwen-3 4B backbone, making the recipe practical on A100 (40GB)-based HPC clusters with low inter-node bandwidth. To better match the training distribution, we replace and adapt the tokenizer, achieving substantially improved tokenization fertility for Hungarian while maintaining competitive performance in English and German. The model is trained on 160B subword tokens drawn from a mixture of internet and high-quality curated sources, with a composition of 44% Hungarian, 24% English, 21% German, and 11% code. This data mix is chosen to mitigate catastrophic forgetting and preserve high-resource language capabilities during continual pretraining. Our preliminary results indicate modest but stable results in language adaptation.
Paper Structure (21 sections, 1 figure, 5 tables, 1 algorithm)

This paper contains 21 sections, 1 figure, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. The Challenge of Linguistic Digital Sovereignty for Medium-Resource Languages
  3. Core Contributions
  4. Related Work
  5. Paradigms of Language Model Adaptation
  6. The Landscape of Hungarian Language Models
  7. Training Data Curation
  8. Corpus Design and Composition
  9. Data Sources
  10. Methodology
  11. Backbone Model Architecture
  12. Tokenizer Adaptation for Hungarian
  13. Packed Continual Pretraining with LoRA
  14. Training Infrastructure and Parallelization Strategy
  15. Evaluation Protocol
  16. ...and 6 more sections

Figures (1)

  • Figure 1: Average training Categorical-Crossentropy loss logged every $200$ steps (left) and validation perplexity measured every $12\ 000$ steps on a language-stratified held out set of documents (right).