Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SiPaKosa: A Comprehensive Corpus of Canonical and Classical Buddhist Texts in Sinhala and Pali

Ranidu Gurusinghe, Nevidu Jayatilleke

Abstract

SiPaKosa is a comprehensive corpus of Sinhala and Pali doctrinal texts comprising approximately 786K sentences and 9.25M words, incorporating 16 copyright-cleared historical Buddhist documents alongside the complete web-scraped Tripitaka canonical texts. The corpus was created through high-quality OCR using Google Document AI on historical manuscripts, combined with systematic web scraping of canonical repositories, followed by rigorous quality control and metadata annotation. The corpus is organised into language-specific subcorpora: Sinhala and Mixed Sinhala-Pali. We evaluate the performance of language models using ten pretrained models, with perplexity scores ranging from 1.09 to 189.67 on our corpus. This analysis shows that proprietary models significantly outperform open-source alternatives by factors of three to six times. This corpus supports the pretraining of domain-adapted language models, facilitates historical language analysis, and aids in the development of information retrieval systems for Buddhist scholarship while preserving Sinhala cultural heritage.

SiPaKosa: A Comprehensive Corpus of Canonical and Classical Buddhist Texts in Sinhala and Pali

Abstract

SiPaKosa is a comprehensive corpus of Sinhala and Pali doctrinal texts comprising approximately 786K sentences and 9.25M words, incorporating 16 copyright-cleared historical Buddhist documents alongside the complete web-scraped Tripitaka canonical texts. The corpus was created through high-quality OCR using Google Document AI on historical manuscripts, combined with systematic web scraping of canonical repositories, followed by rigorous quality control and metadata annotation. The corpus is organised into language-specific subcorpora: Sinhala and Mixed Sinhala-Pali. We evaluate the performance of language models using ten pretrained models, with perplexity scores ranging from 1.09 to 189.67 on our corpus. This analysis shows that proprietary models significantly outperform open-source alternatives by factors of three to six times. This corpus supports the pretraining of domain-adapted language models, facilitates historical language analysis, and aids in the development of information retrieval systems for Buddhist scholarship while preserving Sinhala cultural heritage.

Paper Structure

This paper contains 41 sections, 3 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Bible in 100 languages
  4. Sinhala Diachronic Corpus
  5. Religious and Cultural Text Corpora
  6. Methodology
  7. Data Collection
  8. Text Extraction and Processing
  9. Language Classification
  10. Corpus Integration and Metadata
  11. Evaluation of SiPaKosa
  12. Corpus Statistics
  13. Language Distribution
  14. Results
  15. Evaluation Setup
  16. ...and 26 more sections

Figures (5)

  • Figure 1: Complete methodology pipeline showing dual-source data collection, processing, and integration.
  • Figure 2: Book-level filtering from 83 downloaded books to 16 corpus-eligible books.
  • Figure 3: Sentence distribution by language and source.
  • Figure 4: PDF processing pipeline applied to the 16 copyright-cleared historical Buddhist books, combining pdfplumber text extraction and Google Document AI OCR, with copyright checking, page classification, and confidence-based quality filtering (threshold: 0.7) prior to corpus integration.
  • Figure 5: JSON structure of a metadata record of the books in the IFBC corpus, defining the fields and their corresponding data types.