TF3-RO-50M: Training Compact Romanian Language Models from Scratch on Synthetic Moral Microfiction

TL;DR

TF3-RO-50M demonstrates that a Romanian language model trained entirely from synthetic moral microfiction can reach stable optimization and coherent morphosyntactic behavior at a compact size. The approach couples Romanian-specific tokenization (32k Unigram), long-context 2{,}048-token packing, and from-scratch pretraining of a $51.65{,}\mathrm{M}$-parameter Transformer, followed by pruning and logit-based distillation to a $26.45{,}\mathrm{M}$-parameter student. Compression via quantization and distillation yields a practical deployment frontier, with trade-offs between efficiency and linguistic quality clearly characterized through a multi-faceted evaluation suite. The work provides a fully reproducible, end-to-end pipeline for Romanian NLP that leverages synthetic data while highlighting the importance of tokenizer design and multi-dimensional assessment for morphologically rich languages. Overall, TF3-RO advances compact Romanian LMs by integrating tokenizer design, synthetic data generation, structured pruning, distillation, and robust evaluation into a single, reproducible framework with immediate deployment utility.

Abstract

Recent advances in synthetic data generation have shown that compact language models can be trained effectively when the underlying corpus is structurally controlled and linguistically coherent. However, for morphologically rich and computationally under-resourced languages such as Romanian, there is still no openly documented, end-to-end pipeline that unifies tokenizer design, preprocessing, pretraining, compression, evaluation, and large-scale synthetic data generation in a reproducible framework. Building on TF1, a three-million-story English fable dataset, and TF2, which extends TF1 through high-quality Romanian translations, we introduce TF3-RO, a Romanian-centric language modeling pipeline spanning tokenizer training, from-scratch model development, and Romanian-native dataset generation. TF3-RO constructs Romanian-specific BPE and Unigram tokenizers from a linguistically informed corpus to mitigate token inflation induced by Romanian morphology. Using long-sequence packed training, we pretrain a 51.65M-parameter LLaMA-style Transformer entirely from scratch. The model is subsequently optimized through quantization, structured pruning, and logit-based knowledge distillation, yielding a compact 26.45M-parameter student model with tied embeddings and strong deployment characteristics. Using this distilled model, TF3-RO generates three million Romanian-native synthetic fables via a controlled combinatorial prompting framework. Across all stages, the pipeline integrates a comprehensive evaluation suite combining intrinsic metrics, Romanian agreement probes, entity coherence, rule-based grammar checking, and LLM-based assessment. TF3-RO provides a reproducible and linguistically grounded framework for training compact Romanian language models and producing large-scale synthetic narrative corpora.

Figures (2)

• Figure 1: Training evolution of the 51.65M Transformer on TF3-RO-50M: cross-entropy (CE) and perplexity (PPL) over training steps.
• Figure 2: Evaluation of five TF3 models---Transformer (Teacher), Distilled Transformer (Student), Transformer-Q8, Transformer-Q6, and Mamba---across model size, intrinsic metrics, entity coherence, throughput, rule-based grammar, and LLM-based evaluation.