Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi

Shiza Fatimah, Aniket Sen, Sophia Falk, Florian Mai, Lucie Flek, Nicholas Kluge Corrêa

TL;DR

Across comprehensive evaluation suites, LilMoo consistently outperforms comparably sized multilingual baselines such as Qwen2.5-0.5B and Qwen3-0.6B, demonstrating that well-designed language-specific pretraining can rival large multilingual models at the sub-billion-parameter range.

Abstract

The dominance of large multilingual foundation models has widened linguistic inequalities in Natural Language Processing (NLP), often leaving low-resource languages underrepresented. This paper introduces LilMoo, a 0.6-billion-parameter Hindi language model trained entirely from scratch to address this gap. Unlike prior Hindi models that rely on continual pretraining from opaque multilingual foundations, LilMoo is developed through a fully transparent and reproducible pipeline optimized for limited compute environments. We construct a high-quality Hindi corpus (GigaLekh) filtered through both heuristic and learned (LLM-as-a-judge) methods, complemented by bilingual augmentation with curated English data. Using this dataset, we explore various training recipes for small-scale language models. Across comprehensive evaluation suites, LilMoo consistently outperforms comparably sized multilingual baselines such as Qwen2.5-0.5B and Qwen3-0.6B, demonstrating that well-designed language-specific pretraining can rival large multilingual models at the sub-billion-parameter range.

Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi

TL;DR

Across comprehensive evaluation suites, LilMoo consistently outperforms comparably sized multilingual baselines such as Qwen2.5-0.5B and Qwen3-0.6B, demonstrating that well-designed language-specific pretraining can rival large multilingual models at the sub-billion-parameter range.

Abstract

The dominance of large multilingual foundation models has widened linguistic inequalities in Natural Language Processing (NLP), often leaving low-resource languages underrepresented. This paper introduces LilMoo, a 0.6-billion-parameter Hindi language model trained entirely from scratch to address this gap. Unlike prior Hindi models that rely on continual pretraining from opaque multilingual foundations, LilMoo is developed through a fully transparent and reproducible pipeline optimized for limited compute environments. We construct a high-quality Hindi corpus (GigaLekh) filtered through both heuristic and learned (LLM-as-a-judge) methods, complemented by bilingual augmentation with curated English data. Using this dataset, we explore various training recipes for small-scale language models. Across comprehensive evaluation suites, LilMoo consistently outperforms comparably sized multilingual baselines such as Qwen2.5-0.5B and Qwen3-0.6B, demonstrating that well-designed language-specific pretraining can rival large multilingual models at the sub-billion-parameter range.
Paper Structure (59 sections, 4 equations, 8 figures, 15 tables)

This paper contains 59 sections, 4 equations, 8 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Hindi Language Models
  4. Challenges and Motivation
  5. Pretraining Data
  6. Corpus Construction Pipeline
  7. Stage 1: Source Screening and Text Extraction
  8. Stage 2: Language Verification
  9. Stage 3: Heuristic Content Filtering
  10. Stage 5: Duplicate Removal
  11. Stage 6: Learned Quality Filtering
  12. Learned Quality Filtering
  13. LLM Annotation
  14. Training Classifiers
  15. Final Dataset Composition
  16. ...and 44 more sections

Figures (8)

  • Figure 1: Pretraining loss curve for the LilMoo pair.
  • Figure 2: Gradient Statistics for the LilMoo pair.
  • Figure 3: Per-benchmark performance across our evaluation suite (Qwen2.5/3 serve as Baselines).
  • Figure 4: NPM scores for both LilMoo models and both Qwen baselines.
  • Figure 5: Performance vs Compute for LilMoo Models vs Qwen Baselines
  • ...and 3 more figures