Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fine-Tuning LLMs for Low-Resource Dialect Translation: The Case of Lebanese

Silvana Yakhni, Ali Chehab

TL;DR

The paper tackles Lebanese Arabic dialect MT under resource scarcity by comparing fine-tuning on culturally authentic LW data against larger non-native corpora. It introduces three instruction types—Translation, Contrastive, and Grammar-Hint—and explores prompting strategies, using Aya23-8B with Qlora for fine-tuning. Key findings show data quality and cultural alignment trump sheer data volume, with contrastive fine-tuning and contrastive prompting yielding strong results, and LebEval revealing substantial differences from generic benchmarks like FloRes. The work underscores the importance of authentic evaluation datasets and proposes avenues for richer dialectal MT, including future work on mixture-of-experts and agentic models to further leverage cultural nuance in translation.

Abstract

This paper examines the effectiveness of Large Language Models (LLMs) in translating the low-resource Lebanese dialect, focusing on the impact of culturally authentic data versus larger translated datasets. We compare three fine-tuning approaches: Basic, contrastive, and grammar-hint tuning, using open-source Aya23 models. Experiments reveal that models fine-tuned on a smaller but culturally aware Lebanese dataset (LW) consistently outperform those trained on larger, non-native data. The best results were achieved through contrastive fine-tuning paired with contrastive prompting, which indicates the benefits of exposing translation models to bad examples. In addition, to ensure authentic evaluation, we introduce LebEval, a new benchmark derived from native Lebanese content, and compare it to the existing FLoRes benchmark. Our findings challenge the "More Data is Better" paradigm and emphasize the crucial role of cultural authenticity in dialectal translation. We made our datasets and code available on Github.

Fine-Tuning LLMs for Low-Resource Dialect Translation: The Case of Lebanese

TL;DR

The paper tackles Lebanese Arabic dialect MT under resource scarcity by comparing fine-tuning on culturally authentic LW data against larger non-native corpora. It introduces three instruction types—Translation, Contrastive, and Grammar-Hint—and explores prompting strategies, using Aya23-8B with Qlora for fine-tuning. Key findings show data quality and cultural alignment trump sheer data volume, with contrastive fine-tuning and contrastive prompting yielding strong results, and LebEval revealing substantial differences from generic benchmarks like FloRes. The work underscores the importance of authentic evaluation datasets and proposes avenues for richer dialectal MT, including future work on mixture-of-experts and agentic models to further leverage cultural nuance in translation.

Abstract

This paper examines the effectiveness of Large Language Models (LLMs) in translating the low-resource Lebanese dialect, focusing on the impact of culturally authentic data versus larger translated datasets. We compare three fine-tuning approaches: Basic, contrastive, and grammar-hint tuning, using open-source Aya23 models. Experiments reveal that models fine-tuned on a smaller but culturally aware Lebanese dataset (LW) consistently outperform those trained on larger, non-native data. The best results were achieved through contrastive fine-tuning paired with contrastive prompting, which indicates the benefits of exposing translation models to bad examples. In addition, to ensure authentic evaluation, we introduce LebEval, a new benchmark derived from native Lebanese content, and compare it to the existing FLoRes benchmark. Our findings challenge the "More Data is Better" paradigm and emphasize the crucial role of cultural authenticity in dialectal translation. We made our datasets and code available on Github.
Paper Structure (19 sections, 7 figures, 2 tables)

This paper contains 19 sections, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Instruction Pool
  3. Translation Instruction
  4. Contrastive Instruction
  5. Grammar-Hint Instruction
  6. Experimental Setup
  7. Training Data
  8. Model Training
  9. Prompting
  10. Evaluation
  11. Results
  12. Ablation Study
  13. Main Results
  14. Preference Alignment vs. Fine-tuning
  15. Conclusion
  16. ...and 4 more sections

Figures (7)

  • Figure 1: Example of the translation of a cultural Lebanese idiom by a human translatorcompared to GPT-4o
  • Figure 2: Translation Instructions Templates
  • Figure 3: Illustration of four single-step and Curriculum Training Configurations
  • Figure 4: Impact of few-shot example selection methods (random, embedding-based, and frequency-based matching) and varying K values (K=3, 5, 7) on the translation quality of Aya23-8B.
  • Figure 5: Steps performed to synthesize the Lebanese Grammatical Data: 1) Choosing a Lebanese Grammatical book, 2) Chunking the book into small Grammatical paragraphs, 3) Prompting Claude to use the small paragraphs, to 4) generate Grammatical instructions.
  • ...and 2 more figures