Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IndicSuperTokenizer: An Optimized Tokenizer for Indic Multilingual LLMs

Souvik Rana, Arul Menezes, Ashish Kulkarni, Chandra Khatri, Shubham Agarwal

TL;DR

IndicSuperTokenizer tackles tokenization inefficiencies in multilingual Indic LLMs by introducing a two-stage curriculum that learns subword units first and then multiword units, guided by language-aware pre-tokenization. The approach yields state-of-the-art fertility across 22 Indic languages, English, and code, while delivering notable throughput gains and maintaining strong benchmark performance. Extensive intrinsic and extrinsic evaluations, including ablations on data size, transition point, vocabulary size, and normalization, demonstrate robustness and practical deployment benefits. The work also demonstrates corpus-driven vocabulary allocation and the feasibility of replacing opensource tokenizers via continual pretraining, highlighting IST's applicability to open multilingual models.

Abstract

Tokenizers play a crucial role in determining the performance, training efficiency, and the inference cost of Large Language Models (LLMs). Designing effective tokenizers for multilingual LLMs is particularly challenging due to diverse scripts and rich morphological variation. While subword methods such as Byte Pair Encoding (BPE) are widely adopted, their effectiveness in multilingual settings remains underexplored. We present IndicSuperTokenizer, a tokenizer for Indic multilingual LLMs, that combines both subword and multi-word tokenization, along with language-specific pre-tokenization, leading to more linguistically aligned tokens and achieving a new state-of-the-art in fertility score. Evaluated across English, 22 Indian languages and code data, our tokenizer improves the average fertility score by 39.5% over LLaMA4 and by 18% over Sutra (the current best). This translates to 44% improvement in inference throughput over LLaMA4 while maintaining comparable performance on English and Indic benchmarks. We also present detailed ablations across tokenizer training data size, vocabulary size, merging techniques, and pre-tokenization strategies, demonstrating the robustness of our design choices.

IndicSuperTokenizer: An Optimized Tokenizer for Indic Multilingual LLMs

TL;DR

IndicSuperTokenizer tackles tokenization inefficiencies in multilingual Indic LLMs by introducing a two-stage curriculum that learns subword units first and then multiword units, guided by language-aware pre-tokenization. The approach yields state-of-the-art fertility across 22 Indic languages, English, and code, while delivering notable throughput gains and maintaining strong benchmark performance. Extensive intrinsic and extrinsic evaluations, including ablations on data size, transition point, vocabulary size, and normalization, demonstrate robustness and practical deployment benefits. The work also demonstrates corpus-driven vocabulary allocation and the feasibility of replacing opensource tokenizers via continual pretraining, highlighting IST's applicability to open multilingual models.

Abstract

Tokenizers play a crucial role in determining the performance, training efficiency, and the inference cost of Large Language Models (LLMs). Designing effective tokenizers for multilingual LLMs is particularly challenging due to diverse scripts and rich morphological variation. While subword methods such as Byte Pair Encoding (BPE) are widely adopted, their effectiveness in multilingual settings remains underexplored. We present IndicSuperTokenizer, a tokenizer for Indic multilingual LLMs, that combines both subword and multi-word tokenization, along with language-specific pre-tokenization, leading to more linguistically aligned tokens and achieving a new state-of-the-art in fertility score. Evaluated across English, 22 Indian languages and code data, our tokenizer improves the average fertility score by 39.5% over LLaMA4 and by 18% over Sutra (the current best). This translates to 44% improvement in inference throughput over LLaMA4 while maintaining comparable performance on English and Indic benchmarks. We also present detailed ablations across tokenizer training data size, vocabulary size, merging techniques, and pre-tokenization strategies, demonstrating the robustness of our design choices.

Paper Structure

This paper contains 43 sections, 5 figures, 22 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Tokenization Algorithms.
  4. Multilingual Tokenizers.
  5. Pre-tokenization.
  6. IndicSuperTokenizer (IST)
  7. Tokenizer Training
  8. Pre-tokenization
  9. Training data and Vocabulary
  10. Baselines
  11. Metrics
  12. Evaluation Framework
  13. Experiments and Results
  14. Intrinsic Evaluation of Tokenizers
  15. Extrinsic Evaluation on Downstream Tasks
  16. ...and 28 more sections

Figures (5)

  • Figure 1: IndicSuperTokenizer (IST) captures superwords (e.g. "wake up", "in the morning") and avoids fragmenting Indic words (see for e.g. Bengali, Tamil).
  • Figure 2: Vocabulary size distribution across language scripts. See Appendix \ref{['app:iso']} for script details.
  • Figure 3: Tokenized output of morph-aware tokenizer
  • Figure 4: Tokenized output of non morph-aware tokenizer
  • Figure 5: Trend of potential glitch tokens in upper 20K of vocabulary for different K.