SiDiaC: Sinhala Diachronic Corpus
Nevidu Jayatilleke, Nisansa de Silva
TL;DR
SiDiaC introduces the first diachronic Sinhala corpus, spanning roughly 426 CE to 1944 CE, built from 46 books sourced from the National Library of Sri Lanka under copyright and accessibility constraints. The authors employ Google Document AI for OCR, followed by extensive manual post-processing and a two-level genre classification to create rich metadata in alignment with established corpora practices. The evaluation demonstrates strong OCR performance, meaningful metadata coverage across centuries, and token-level analyses that reveal diachronic patterns in vocabulary, stopwords, and script modernization. This resource lays the groundwork for lexical change, neologism tracking, and historical syntax studies in Sinhala, with clear directions for future expansion, code-mixed data handling, and lexical annotations.
Abstract
SiDiaC, the first comprehensive Sinhala Diachronic Corpus, covers a historical span from the 5th to the 20th century CE. SiDiaC comprises 58k words across 46 literary works, annotated carefully based on the written date, after filtering based on availability, authorship, copyright compliance, and data attribution. Texts from the National Library of Sri Lanka were digitised using Google Document AI OCR, followed by post-processing to correct formatting and modernise the orthography. The construction of SiDiaC was informed by practices from other corpora, such as FarPaHC, particularly in syntactic annotation and text normalisation strategies, due to the shared characteristics of low-resourced language status. This corpus is categorised based on genres into two layers: primary and secondary. Primary categorisation is binary, classifying each book into Non-Fiction or Fiction, while the secondary categorisation is more specific, grouping texts under Religious, History, Poetry, Language, and Medical genres. Despite challenges including limited access to rare texts and reliance on secondary date sources, SiDiaC serves as a foundational resource for Sinhala NLP, significantly extending the resources available for Sinhala, enabling diachronic studies in lexical change, neologism tracking, historical syntax, and corpus-based lexicography.