Low-Resource Dialect Adaptation of Large Language Models: A French Dialect Case-Study

TL;DR

This study tackles the dialect gap in large language models by applying compute-efficient continual pre-training with LoRA to adapt three LLMs to Québec French using a compact unlabeled corpus (~86.57M tokens). The authors demonstrate that parameter-efficient CPT can achieve substantial Québec French gains on the COLE benchmark while largely preserving prestige French performance, though success is contingent on model size and corpus composition. Key contributions include a practical CPT pipeline with minimal parameter updates, release of Québec French LLMs on HuggingFace, and detailed analysis of acquisition vs. retention dynamics across tasks. The work has practical implications for expanding equitable AI access to minority language communities and highlights the importance of data composition and model capacity in dialect adaptation.

Abstract

Despite the widespread adoption of large language models (LLMs), their strongest capabilities remain largely confined to a small number of high-resource languages for which there is abundant training data. Recently, continual pre-training (CPT) has emerged as a means to fine-tune these models to low-resource regional dialects. In this paper, we study the use of CPT for dialect learning under tight data and compute budgets. Using low-rank adaptation (LoRA) and compute-efficient continual pre-training, we adapt three LLMs to the Québec French dialect using a very small dataset and benchmark them on the COLE suite. Our experiments demonstrate an improvement on the minority dialect benchmarks with minimal regression on the prestige language benchmarks with under 1% of model parameters updated. Analysis of the results demonstrate that gains are highly contingent on corpus composition. These findings indicate that CPT with parameter-efficient fine-tuning (PEFT) can narrow the dialect gap by providing cost-effective and sustainable language resource creation, expanding high-quality LLM access to minority linguistic communities. We release the first Québec French LLMs on HuggingFace.

Figures (1)

• Figure 1: Perplexity during CPT training. Lower perplexity indicates better fit to Québec French. All models evaluated on identical held-out corpus.