ks-lit-3m: A 3.1 million word kashmiri text dataset for large language model pretraining

TL;DR

Kashmiri language processing suffers from data scarcity and an InPage legacy format that hides high-quality text from modern NLP pipelines. The authors introduce KS-LIT-3M, a 3.1 million word Kashmiri pretraining corpus produced via an InPage-to-Unicode converter, with careful preprocessing to remove English contamination, normalize characters, and validate quality. The dataset spans diverse genres and decades, preserving diacritics and providing CC-BY-4.0 licensing to enable broad research and development in Kashmiri NLP. This resource enables more accurate Kashmiri language model pretraining, supports downstream NLP tasks, and offers a template for recovering other languages affected by legacy formats.

Abstract

Large Language Models (LLMs) demonstrate remarkable fluency across high-resource languages yet consistently fail to generate coherent text in Kashmiri, a language spoken by approximately seven million people. This performance disparity stems not from inherent model limitations but from a critical scarcity of high-quality training data. Decades of Kashmiri literature remain inaccessible to modern NLP pipelines due to their encoding in the proprietary InPage desktop publishing format. This paper introduces KS-LIT-3M, a curated corpus of 3.1 million words (16.4 million characters) specifically designed for pretraining language models on Kashmiri. The dataset is structured as a single continuous linear text stream, optimized for causal language model training where models learn to predict subsequent tokens from preceding context. The corpus was constructed through the development of a specialized InPage-to-Unicode converter, followed by rigorous preprocessing including English contamination removal, character normalization, and quality validation. Encompassing 131,607 unique words drawn from diverse genres including literary works, journalistic writing, academic texts, and religious scholarship, KS-LIT-3M addresses a fundamental resource gap for Kashmiri language technology. The dataset is released under the CC-BY-4.0 license to facilitate research in Kashmiri natural language processing.