Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages

S. Tamang, D. J. Bora

TL;DR

The findings reveal that the SUTRA tokenizer outperforms all other models, including several Indic-specific models, excelling in 14 languages, and lays the groundwork for future improvements in tokenizer design to enhance linguistic coverage and model efficiency.

Abstract

Large Language Models (LLMs) based on transformer architectures have revolutionized a variety of domains, with tokenization playing a pivotal role in their pre-processing and fine-tuning stages. In multilingual models, particularly those tailored for Indic languages, effective tokenization is crucial for optimizing performance. This paper presents a comprehensive evaluation of tokenizers used by 12 LLMs across all 22 official languages of India, with a focus on comparing the efficiency of their tokenization processes. We employed the Normalized Sequence Length (NSL) as a key metric in our analysis. Our findings reveal that the SUTRA tokenizer outperforms all other models, including several Indic-specific models, excelling in 14 languages. Notable insights include the SUTRA tokenizer's superior handling of Indic languages, GPT-4o's advancement over its predecessor GPT-4 in processing Indian languages, and the limited performance of Project Indus in certain languages. This study underscores the critical importance of developing targeted tokenization strategies for multilingual and Indic-centric models, laying the groundwork for future improvements in tokenizer design to enhance linguistic coverage and model efficiency.

Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages

TL;DR

The findings reveal that the SUTRA tokenizer outperforms all other models, including several Indic-specific models, excelling in 14 languages, and lays the groundwork for future improvements in tokenizer design to enhance linguistic coverage and model efficiency.

Abstract

Large Language Models (LLMs) based on transformer architectures have revolutionized a variety of domains, with tokenization playing a pivotal role in their pre-processing and fine-tuning stages. In multilingual models, particularly those tailored for Indic languages, effective tokenization is crucial for optimizing performance. This paper presents a comprehensive evaluation of tokenizers used by 12 LLMs across all 22 official languages of India, with a focus on comparing the efficiency of their tokenization processes. We employed the Normalized Sequence Length (NSL) as a key metric in our analysis. Our findings reveal that the SUTRA tokenizer outperforms all other models, including several Indic-specific models, excelling in 14 languages. Notable insights include the SUTRA tokenizer's superior handling of Indic languages, GPT-4o's advancement over its predecessor GPT-4 in processing Indian languages, and the limited performance of Project Indus in certain languages. This study underscores the critical importance of developing targeted tokenization strategies for multilingual and Indic-centric models, laying the groundwork for future improvements in tokenizer design to enhance linguistic coverage and model efficiency.

Paper Structure

This paper contains 24 sections, 1 equation, 27 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Tokenizers in Multilingual & Indic Models
  4. Literature Review
  5. Large Language Models and Tokenization
  6. Indic-Specific Language Models
  7. Comparative Evaluation of Tokenizers
  8. Gaps in Existing Research
  9. Methodology
  10. Example Texts
  11. Models
  12. Evaluation Metric
  13. Results
  14. Discussion
  15. Microsoft's Phi-3.5-MoE-instruct and Google's Indic Gemma
  16. ...and 9 more sections

Figures (27)

  • Figure 1: Evaluation pipeline: (1) We collect example texts for all 22 languages. (2) We send the example texts to the LLMs' tokenizer. (3) Evaluate the tokenized outputs. (4) We construct leaderboards using our evaluation.
  • Figure 2: Assamese text used for evaluating tokenizer performance.
  • Figure 3: Number of Best Performances Achieved by Each Tokenizer Across 22 Languages.
  • Figure 4: Number of tokens required for a single example text in Assamese. Lower values are better.
  • Figure 5: Number of tokens required for a single example text in Bengali. Lower values are better.
  • ...and 22 more figures