Adapting Large Language Models to Low-Resource Tibetan: A Two-Stage Continual and Supervised Fine-Tuning Study

TL;DR

This study tackles the challenge of adapting a large decoder-only model to Tibetan under data scarcity by a two-stage pipeline: Continual Pretraining to ground Tibetan linguistics, followed by Supervised Fine-Tuning for task and translation specialization. The approach yields monotonic improvements in perplexity and translation quality, with Chinese→Tibetan BLEU improving over 5x and chrF over 3x, and shows that adaptation concentrates in embeddings, output heads, and mid-to-late MLPs. Layer-wise analysis reveals near-perfect alignment between CPT and SFT weight changes, indicating consolidation of the Tibetan semantic manifold rather than overwriting prior representations. The work offers a reproducible framework for extending multilingual foundation models to low-resource languages and provides insights into how such models internalize language-specific structure without catastrophic forgetting.

Abstract

Adapting large language models (LLMs) to low-resource languages remains a major challenge due to data scarcity and cross-lingual drift. This work presents a two-stage adaptation of Qwen2.5-3B to Tibetan, a morphologically rich and underrepresented language. We employ Continual Pretraining (CPT) to establish Tibetan linguistic grounding, followed by Supervised Fine-Tuning (SFT) for task and translation specialization. Empirical evaluations demonstrate a consistent decrease in perplexity (from 2.98 $\rightarrow$ 1.54) and substantial improvements in Chinese$\rightarrow$Tibetan translation quality (BLEU: 0.046 $\rightarrow$ 0.261; chrF: 2.2 $\rightarrow$ 6.6). Layer-wise analysis across 435 layers in Qwen3-4B reveals that adaptation primarily concentrates on embedding and output heads, with mid--late MLP projections encoding domain-specific transformations. Our findings suggest that CPT constructs a Tibetan semantic manifold while SFT sharpens task alignment with minimal representational disruption. This study provides the first quantitative exploration of Tibetan adaptation dynamics for LLMs, and offers an open, reproducible framework for extending multilingual foundation models to low-resource settings.

Figures (9)

• Figure 1: Performance metrics overview showing consistent improvements from Base through CPT to SFT across language modeling and translation tasks.
• Figure 2: Two-stage training pipeline and data composition. Continual pretraining (CPT) uses Tibetan monolingual text derived from parallel corpora and non-parallel sources; supervised fine-tuning (SFT) uses an instruction mixture (BO→BO, CN→BO, EN→BO) with a small general Chinese component for multilingual anchoring.
• Figure 3: Translation quality progression across stages. Chinese→Tibetan exhibits the largest improvements due to explicit supervision in SFT, while English→Tibetan benefits from cross-lingual transfer.
• Figure 4: chrF score comparison highlighting consistent gains from Base to SFT for both translation directions.
• Figure 5: Top layer changes during CPT. Embeddings, output head, and mid-late MLP gate projections (layers 21-23) dominate.