Krutrim LLM: A Novel Tokenization Strategy for Multilingual Indic Languages with Petabyte-Scale Data Processing

TL;DR

The paper tackles the scarcity of high-quality multilingual Indic data and tokenizer design for Indic LLMs by constructing a data pipeline over petabyte-scale sources. It integrates Common Crawl processing with language detection, deduplication via Minhash LSH, and structured extraction from newspapers and books using advanced computer vision and OCR techniques to produce a rich 12-language Indic corpus. A novel multilingual tokenizer training strategy yields a superior token-to-word ratio compared to OpenAI's Tiktoken, validated across 11 Indic languages, supporting efficient 3B- and 7B-parameter Indic LLMs. Overall, the work demonstrates how careful data curation and language-specific tokenization can enable more accurate and efficient Indic-language models with broad practical impact for multilingual NLP in India.

Abstract

We present a novel approach to data preparation for developing multilingual Indic large language model. Our meticulous data acquisition spans open-source and proprietary sources, including Common Crawl, Indic books, news articles, and Wikipedia, ensuring a diverse and rich linguistic representation. For each Indic language, we design a custom preprocessing pipeline to effectively eliminate redundant and low-quality text content. Additionally, we perform deduplication on Common Crawl data to address the redundancy present in 70% of the crawled web pages. This study focuses on developing high-quality data, optimizing tokenization for our multilingual dataset for Indic large language models with 3B and 7B parameters, engineered for superior performance in Indic languages. We introduce a novel multilingual tokenizer training strategy, demonstrating our custom-trained Indic tokenizer outperforms the state-of-the-art OpenAI Tiktoken tokenizer, achieving a superior token-to-word ratio for Indic languages.

Figures (1)

• Figure 1: Common Crawl Dataset Processing Pipeline