Mecellem Models: Turkish Models Trained from Scratch and Continually Pre-trained for the Legal Domain

TL;DR

Mecellem tackles Turkish legal NLP by pairing encoder models pre-trained from scratch with a large Turkish-law corpus and decoder models adapted through continual pre-training. A key finding is that optimal downstream utility often occurs at intermediate checkpoints rather than MLM-loss minima, particularly for morphologically rich Turkish; this drives the design of a checkpoint-aware pre-training strategy and a four-phase CPT curriculum. The encoder approach achieves top Turkish retrieval performance with strong parameter efficiency, while CPT improves long-context legal understanding in decoder models, with perplexity reductions up to 36.2% and competitive downstream benchmarks. Collectively, Mecellem demonstrates practical, resource-conscious pathways to domain-specific Turkish legal LMs suitable for retrieval and RAG-like applications, with open-source release to support reproducibility and adoption.

Abstract

This paper presents Mecellem models, a framework for developing specialized language models for the Turkish legal domain through domain adaptation strategies. We make two contributions: (1)Encoder Model Pre-trained from Scratch: ModernBERT-based bidirectional encoders pre-trained on a Turkish-dominant corpus of 112.7 billion tokens. We implement a checkpoint selection strategy that evaluates downstream retrieval performance throughout training, revealing that optimal checkpoints achieve best retrieval scores before pre-training loss reaches its minimum. Our encoder models achieve top-3 rankings on the Turkish retrieval leaderboard, with smaller models (155M parameters) achieving comparable performance to larger reference models (307M-567M parameters). Our approach achieves 92.36% production efficiency compared to state-of-the-art models (embeddinggemma-300m: 100.00%, BAAI/bge-m3: 99.54%, newmindai/bge-m3-stsb: 94.38%), ranking fourth overall despite requiring less computational resources. SOTA models rely on multi-stage, computationally intensive training pipelines, making our single-stage pre-training followed by efficient post-training approach a cost-effective alternative; (2)Decoder Model with Continual Pre-training (CPT): Qwen3-1.7B and Qwen3-4B models adapted to Turkish legal domain through controlled curriculum learning. Four-phase CPT with optimal sample ratios enables gradual transition from general language knowledge to specialized legal terminology and long-context reasoning. This approach achieves 36.2% perplexity reduction on Turkish legal text, demonstrating domain adaptation gains.

Figures (16)

• Figure 1: Natural completion rate over a 6.5-hour extraction run.
• Figure 2: Token Count Distribution Analysis Across All Threshold Combinations.
• Figure 3: Token Count Distribution Across Suffix Entropy Thresholds (Lemma Diversity $\ge$ 50%)
• Figure 4: Qwen3-1.7B CPT Dataset Distribution across Four Phases.
• Figure 5: Qwen3-4B CPT Dataset Distribution Single Phase.