Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Regional Tiny Stories: Using Small Models to Compare Language Learning and Tokenizer Performance

Nirvan Patil, Malhar Abhay Inamdar, Agnivo Gosai, Guruprasad Pathak, Anish Joshi, Aryan Sagavekar, Anish Joshirao, Raj Dandekar, Rajat Dandekar, Sreedath Panat

TL;DR

Regional TinyStories extends the English TinyStories paradigm to Hindi, Marathi, and Bengali, investigating how small regional language models learn and how tokenizer design affects performance. The study combines translated TinyStories with large-scale synthetic data generated by LLMs and evaluates using language-specific tokenizers (Sarvam, SUTRA) against a general tokenizer (Tiktoken), with GPT-4o serving as the judgment standard. Key contributions include a dataset of roughly $ oughly 10$M synthetic and translated stories, an information-theoretic and morphological tokenizer analysis (Rényi entropy, MorphScore), and evidence that synthetic data and regionally tailored tokenizers yield superior inference quality at modest model scales (around $54$M parameters) compared with translations and baselines. The work provides practical guidance for efficient, high-quality Indian-language generation and introduces a framework for tokenizer evaluation that accounts for language-specific morphology and entropy, with broad implications for democratizing access to language technology in underserved languages.

Abstract

Small Language Models (SLMs) offer efficient alternatives to LLMs for specific domains. The 2023 TinyStories study developed an English dataset that allows SLMs with 1 to 10 million parameters to produce coherent outputs. Our research expands this framework by translating the original dataset into Indian languages and creating synthetic data using LLMs. We focus on Hindi, Marathi, and Bengali, evaluating SLMs for regional language processing and understanding linguistic complexity. We show that SLMs efficiently process regional languages with significantly fewer parameters than LLMs, providing a complementary framework for ``inference based evaluation" of tokenization strategies and linguistic complexity. Our analysis shows that language-specific tokenizers outperform general-purpose ones for Indian languages. Empirical validations, supported by information-theoretic and morphological analyses, provides fundamental understanding behind the better performance of Hindi models over Marathi and Bengali. Additionally, we show that synthetic datasets outperform translated content for training SLMs. Correlation analyses reveal cross-linguistic patterns and language-specific relationships between creativity, grammatical precision, and narrative completeness. These findings advance both the practical application of SLMs to underserved languages and our theoretical understanding of neural language development.

Regional Tiny Stories: Using Small Models to Compare Language Learning and Tokenizer Performance

TL;DR

Regional TinyStories extends the English TinyStories paradigm to Hindi, Marathi, and Bengali, investigating how small regional language models learn and how tokenizer design affects performance. The study combines translated TinyStories with large-scale synthetic data generated by LLMs and evaluates using language-specific tokenizers (Sarvam, SUTRA) against a general tokenizer (Tiktoken), with GPT-4o serving as the judgment standard. Key contributions include a dataset of roughly M synthetic and translated stories, an information-theoretic and morphological tokenizer analysis (Rényi entropy, MorphScore), and evidence that synthetic data and regionally tailored tokenizers yield superior inference quality at modest model scales (around M parameters) compared with translations and baselines. The work provides practical guidance for efficient, high-quality Indian-language generation and introduces a framework for tokenizer evaluation that accounts for language-specific morphology and entropy, with broad implications for democratizing access to language technology in underserved languages.

Abstract

Small Language Models (SLMs) offer efficient alternatives to LLMs for specific domains. The 2023 TinyStories study developed an English dataset that allows SLMs with 1 to 10 million parameters to produce coherent outputs. Our research expands this framework by translating the original dataset into Indian languages and creating synthetic data using LLMs. We focus on Hindi, Marathi, and Bengali, evaluating SLMs for regional language processing and understanding linguistic complexity. We show that SLMs efficiently process regional languages with significantly fewer parameters than LLMs, providing a complementary framework for ``inference based evaluation" of tokenization strategies and linguistic complexity. Our analysis shows that language-specific tokenizers outperform general-purpose ones for Indian languages. Empirical validations, supported by information-theoretic and morphological analyses, provides fundamental understanding behind the better performance of Hindi models over Marathi and Bengali. Additionally, we show that synthetic datasets outperform translated content for training SLMs. Correlation analyses reveal cross-linguistic patterns and language-specific relationships between creativity, grammatical precision, and narrative completeness. These findings advance both the practical application of SLMs to underserved languages and our theoretical understanding of neural language development.

Paper Structure

This paper contains 69 sections, 27 figures, 17 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methodology: Data generation, training and evaluation experiments
  3. Training data preparation
  4. Translated data
  5. Synthetic data
  6. Training data evaluation
  7. Tokenizer, model and inference evaluation
  8. Results
  9. Insights from our evaluation method
  10. Comparing inference results of our Regional TinyStories with LLMs
  11. Regional tokenizers perform better
  12. Models trained on translated data have lower inference evaluations
  13. Tokenizer analysis with focus on language complexity
  14. Information-Theoretic Analysis
  15. Morph Score
  16. ...and 54 more sections

Figures (27)

  • Figure 1: Schematic of model pipeline. (A) Dataset prepared through machine translation as well as generation using LLM, (B) Indic tokenizers are used to preprocess the Indian language stories, (C) A decoder only transformer architecture (nanoGPT) is used to train the model in each language ; Inference is evaluated by LLM on linguistic attributes
  • Figure 2: Regional TinyStories 5M Hindi model generated story, story after prompt is highlighted
  • Figure 3: GPT4o generated Hindi story
  • Figure 4: Percentage Improvement in Contextual Comprehension, Grammatical Accuracy, and Overall Performance from Baseline Scores across Hindi models detailed in Table 1
  • Figure 5: Percentage Improvement in Contextual Comprehension, Grammatical Accuracy, and Overall Performance from Baseline Scores across Marathi models detailed in Table 2
  • ...and 22 more figures