Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Building pre-train LLM Dataset for the INDIC Languages: a case study on Hindi

Shantipriya Parida, Shakshi Panwar, Kusum Lata, Sanskruti Mishra, Sambit Sekhar

TL;DR

This work tackles the scarcity of high-quality Hindi data for pre-training LLMs by constructing a large Hindi corpus of 1.28B tokens from diverse sources (Wikipedia, dialectal datasets, paraphrase corpora, Oscar, and BigScience-XP3All). It details a full pipeline of data collection and preprocessing, culminating in a freely available dataset on Hugging Face designed to support Hindi LLM pre-training and broader Indic-language research. The authors argue that domain coverage, dialect variety, and transliteration handling are essential for robust Hindi NLP, and they outline use cases from pre-training to synthetic data generation and domain-specific fine-tuning. The study aims to catalyze Hindi-focused NLP development and to offer a replicable template for other Indic or low-resource languages, with emphasis on openness and extensibility.

Abstract

Large language models (LLMs) demonstrated transformative capabilities in many applications that require automatically generating responses based on human instruction. However, the major challenge for building LLMs, particularly in Indic languages, is the availability of high-quality data for building foundation LLMs. In this paper, we are proposing a large pre-train dataset in Hindi useful for the Indic language Hindi. We have collected the data span across several domains including major dialects in Hindi. The dataset contains 1.28 billion Hindi tokens. We have explained our pipeline including data collection, pre-processing, and availability for LLM pre-training. The proposed approach can be easily extended to other Indic and low-resource languages and will be available freely for LLM pre-training and LLM research purposes.

Building pre-train LLM Dataset for the INDIC Languages: a case study on Hindi

TL;DR

This work tackles the scarcity of high-quality Hindi data for pre-training LLMs by constructing a large Hindi corpus of 1.28B tokens from diverse sources (Wikipedia, dialectal datasets, paraphrase corpora, Oscar, and BigScience-XP3All). It details a full pipeline of data collection and preprocessing, culminating in a freely available dataset on Hugging Face designed to support Hindi LLM pre-training and broader Indic-language research. The authors argue that domain coverage, dialect variety, and transliteration handling are essential for robust Hindi NLP, and they outline use cases from pre-training to synthetic data generation and domain-specific fine-tuning. The study aims to catalyze Hindi-focused NLP development and to offer a replicable template for other Indic or low-resource languages, with emphasis on openness and extensibility.

Abstract

Large language models (LLMs) demonstrated transformative capabilities in many applications that require automatically generating responses based on human instruction. However, the major challenge for building LLMs, particularly in Indic languages, is the availability of high-quality data for building foundation LLMs. In this paper, we are proposing a large pre-train dataset in Hindi useful for the Indic language Hindi. We have collected the data span across several domains including major dialects in Hindi. The dataset contains 1.28 billion Hindi tokens. We have explained our pipeline including data collection, pre-processing, and availability for LLM pre-training. The proposed approach can be easily extended to other Indic and low-resource languages and will be available freely for LLM pre-training and LLM research purposes.
Paper Structure (17 sections, 3 figures, 1 table)

This paper contains 17 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Literature Survey
  3. Focused Language
  4. Dataset Preparation
  5. Data Collection
  6. Data Processing
  7. Analysis and Discussion
  8. Domain Coverage
  9. Use Cases
  10. Pre-training:
  11. Language Model:
  12. Generating synthetic data:
  13. Domain-Specific Improvement:
  14. Multilingual NLP Research:
  15. Availability
  16. ...and 2 more sections

Figures (3)

  • Figure 1: The process of gathering information from several sources, such as news articles and supplementary texts, etc., and processing it to produce the pre-trained Hindi dataset is depicted in the figure.
  • Figure 2: This offers a thorough breakdown of the dataset statistics, covering measures like sentence and token counts that provide information regarding the scale and variety of the assembled datasets.
  • Figure 3: This uses a pie chart to depict the distribution of distinct corpora collections, highlighting the proportionate representation of various datasets within the total corpus compilation.