Maastricht University at AMIYA: Adapting LLMs for Dialectal Arabic using Fine-tuning and MBR Decoding

TL;DR

Dialectal Arabic generation and translation are addressed by adapting JAIS-2-based LLMs with LoRA adapters trained separately on monolingual and English–Dialect parallel data. Adapter merging via TIES-Merging and dialect-aware MBR decoding with $N=20$ candidates yield outputs with improved dialect fidelity and preserved semantics across Syrian, Moroccan, and Saudi Arabic. The study shows that combining these techniques outperforms single-source fine-tuning and standard decoding, offering a compact, practical framework for robust dialectal generation. Limitations include limited training data, potential gaps in dialect identification metrics, and increased inference time.

Abstract

Large Language Models (LLMs) are becoming increasingly multilingual, supporting hundreds of languages, especially high resource ones. Unfortunately, Dialect variations are still underrepresented due to limited data and linguistic variation. In this work, we adapt a pre-trained LLM to improve dialectal performance. Specifically, we use Low Rank Adaptation (LoRA) fine-tuning on monolingual and English Dialect parallel data, adapter merging and dialect-aware MBR decoding to improve dialectal fidelity generation and translation. Experiments on Syrian, Moroccan, and Saudi Arabic show that merging and MBR improve dialectal fidelity while preserving semantic accuracy. This combination provides a compact and effective framework for robust dialectal Arabic generation.