IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages

TL;DR

IndicLLMSuite tackles the under-representation of 22 Indian languages in open LLM ecosystems by delivering a cohesive, open-source data suite: Sangraha (pre-training data totaling 251B tokens across verified, synthetic, and unverified splits), Setu (a Spark-based data curation pipeline with robust cleaning, LID, and deduplication), and IndicAlign (Instruct and Toxic datasets for safe instruction-following and toxic alignment). The approach blends high-quality human-verified content with large-scale synthetic and translated data, using structure-preserving translation (Setu-Translate) and transliteration (Setu-Transliterate) to maximize usefulness while maintaining document structure. IndicAlign leverages diverse sources (ShareLlama, WikiHow, IndoWordNet, Wiki-Conv/Chat) and synthetic toxicity data (Toxic Matrix) to train safer, more robust Indic LLMs, with attention to licensing and reuse. The work demonstrates significant token-scale growth over prior Indic corpora and provides open pipelines and artifacts to enable community-driven, multilingual LLM development across languages and domains. This blueprint has practical impact by lowering barriers to building fully open, community-supported Indic LLMs and by offering a replicable framework for extending such efforts to other language groups.

Abstract

Despite the considerable advancements in English LLMs, the progress in building comparable models for other languages has been hindered due to the scarcity of tailored resources. Our work aims to bridge this divide by introducing an expansive suite of resources specifically designed for the development of Indic LLMs, covering 22 languages, containing a total of 251B tokens and 74.8M instruction-response pairs. Recognizing the importance of both data quality and quantity, our approach combines highly curated manually verified data, unverified yet valuable data, and synthetic data. We build a clean, open-source pipeline for curating pre-training data from diverse sources, including websites, PDFs, and videos, incorporating best practices for crawling, cleaning, flagging, and deduplication. For instruction-fine tuning, we amalgamate existing Indic datasets, translate/transliterate English datasets into Indian languages, and utilize LLaMa2 and Mixtral models to create conversations grounded in articles from Indian Wikipedia and Wikihow. Additionally, we address toxicity alignment by generating toxic prompts for multiple scenarios and then generate non-toxic responses by feeding these toxic prompts to an aligned LLaMa2 model. We hope that the datasets, tools, and resources released as a part of this work will not only propel the research and development of Indic LLMs but also establish an open-source blueprint for extending such efforts to other languages. The data and other artifacts created as part of this work are released with permissive licenses.

Figures (32)

• Figure 1: Overview of the different components present in IndicLLMSuite.
• Figure 2:
• Figure 3: Overview of Setu, the data cleaning pipeline used for curating the Sangraha Verified corpus
• Figure 4: Example showing noisy content being extracted from the HTML using trafilatura
• Figure 5: Illustrative examples of pages flagged in various bounding box filters.