Arabic Automatic Story Generation with Large Language Models

TL;DR

This work tackles automatic Arabic story generation by fine-tuning AraLLaMA-2 on a combination of translated TinyStories data and GPT-4-synthesized narratives across MSA and two dialects (Egyptian and Moroccan). It compares two training regimes—direct SFT and a two-step process—and evaluates the resulting models with GPT-4 as an evaluator and with native-human judges. The study introduces a dialect-aware prompt template with 12 controllable features, plus a data-filtering pipeline that yields high-quality Arabic stories. The results show competitive performance against strong baselines and provide publicly released datasets and models to spur further research in Arabic storytelling with LLMs. This work advances Arabic NLP by enabling coherent, instruction-following storytelling across dialects and offers a practical framework for data creation, fine-tuning, and evaluation in low-resource languages.

Abstract

Large language models (LLMs) have recently emerged as a powerful tool for a wide range of language generation tasks. Nevertheless, this progress has been slower in Arabic. In this work, we focus on the task of generating stories from LLMs. For our training, we use stories acquired through machine translation (MT) as well as GPT-4. For the MT data, we develop a careful pipeline that ensures we acquire high-quality stories. For our GPT-41 data, we introduce crafted prompts that allow us to generate data well-suited to the Arabic context in both Modern Standard Arabic (MSA) and two Arabic dialects (Egyptian and Moroccan). For example, we generate stories tailored to various Arab countries on a wide host of topics. Our manual evaluation shows that our model fine-tuned on these training datasets can generate coherent stories that adhere to our instructions. We also conduct an extensive automatic and human evaluation comparing our models against state-of-the-art proprietary and open-source models. Our datasets and models will be made publicly available at https: //github.com/UBC-NLP/arastories.

Figures (6)

• Figure 1: Samples of different stories generated with our models for the three Arabic Varieties.
• Figure 2: The prompt we pass to GPT-4-Turbo to evaluate stories generated with different models.
• Figure 3: Models performance across the MSA, Egyptian, and Moroccan varieties.
• Figure 4: Model A vs. other models in MSA human evaluation. Numbers reflect the number of times Model A is preferred over other models.
• Figure 5: Human comparison of Model A vs Model B and Command-R on Moroccan and Egyptian dialectal story generation. the numbers reflect the number of times Model A was preferred over other models in percentage.