HindiLLM: Large Language Model for Hindi
Sanjay Chouhan, Shubha Brata Nath, Aparajita Dutta
TL;DR
This work develops Hindi-focused large language models by creating a dedicated Byte-level BPE tokenizer and training two autoregressive models, HindiLLM-Small and HindiLLM-Medium, through unsupervised pre-training followed by supervised fine-tuning. Using a large Devanagari Hindi corpus with additional Hind-English translation data, the authors demonstrate that the HindiLLM-Medium model achieves superior performance across sentiment analysis, text classification, NLI, and IndicGLUE benchmarks, often surpassing GPT-2 baselines and GPT-3.5 prompt-based variants. The study highlights the importance of language-specific tokenization and substantial pre-training data for Indic languages, and it provides a detailed dataset and training regimen that can guide future Hindi and other Indic LLMs. The results suggest practical value in deploying HindiLLM-Medium for real-world Hindi NLP tasks and point to future improvements through Hinglish integration and larger-scale training.
Abstract
The advancements in the Large Language Model (LLM) have helped in solving several problems related to language processing. Most of the researches have focused on the English language only, because of its popularity and abundance on the internet. However, a high-performance language model for Hindi and other Indic languages is lacking in the literature. In this work, we have pre-trained two autoregressive LLM models for the Hindi language, namely HindiLLM-Small and HindiLLM-Medium. We use a two-step process comprising unsupervised pre-training and supervised fine-tuning. First, we create a large and high-quality text corpus for unsupervised pre-training. Next, we train a Byte-Pair Encoding, named HindiLLM tokenizer, using the pre-training text data. We then perform training on the unlabeled data, known as the pre-training step, to get the HindiLLM base models. Furthermore, we perform fine-tuning of the HindiLLM base models for different tasks like sentiment analysis, text classification, natural language inference, and multiple choice question-answer on popular labeled datasets to measure the real-world performance. The evaluation shows that the HindiLLM-based fine-tuned models outperform several models in most of the language related tasks.