InstructLR: A Scalable Approach to Create Instruction Dataset for Under-Resourced Languages

TL;DR

InstructLR addresses the data scarcity problem for low-resource languages by integrating LLM-driven seed-instruction generation, translation-aware instruction-response pairing, and a dual-layer quality filter (RAG-based automated checks plus human validation) to produce high-quality, domain-diverse instruction datasets. The framework is demonstrated by building three 50k benchmarks (ZarmaInstruct-50k, BambaraInstruct-50k, FulfuldeInstruct-50k) and by showing that finetuning open-source models on InstructLR data yields substantial improvements over zero-shot and machine-translation baselines, with strong human-judged quality and better downstream NER performance. The results support InstructLR as a scalable, language-agnostic solution for instruction-tuning in LRLS, enabling more capable open-source models and broader real-world applicability. The work also provides insights into cost-effective data creation and highlights avenues for reducing dependence on commercial LLMs and expanding to additional languages.

Abstract

Effective text generation and chat interfaces for low-resource languages (LRLs) remain a challenge for state-of-the-art large language models (LLMs) to support. This is mainly due to the difficulty of curating high-quality instruction datasets for LRLs, a limitation prevalent in the languages spoken across the African continent and other regions. Current approaches, such as automated translation and synthetic data generation, frequently yield outputs that lack fluency or even orthographic consistency. In this paper, we introduce InstructLR, a novel framework designed to generate high-quality instruction datasets for LRLs. Our approach integrates LLM-driven text generation with a dual-layer quality filtering mechanism: an automated filtering layer based on retrieval-augmented-generation (RAG)-based n-shot prompting, and a human-in-the-loop validation layer. Drawing inspiration from benchmarks such as MMLU in task definition, InstructLR has facilitated the creation of three multi-domain instruction benchmarks: ZarmaInstruct-50k, BambaraInstruct-50k, and FulfuldeInstruct-50k.

Figures (6)

• Figure 1: The InstructLR pipeline for creating high-quality instruction-tuning datasets for LRLs. The pipeline starts by the topic curation and finishes by final output.
• Figure 2: Prompt used for RAG-based analysis and evaluation.
• Figure 3: Example of RAG analysis output for a single sentence.
• Figure 4: Estimated total project cost for producing 50,000 instruction--response pairs under three quality--control scenarios. Each bar shows the combined LLM compute/API cost and any required human annotation.
• Figure 5: Prompt used for generating seed instructions from a specific domain.