Sri Lanka Document Datasets: A Large-Scale, Multilingual Resource for Law, News, and Policy

TL;DR

Problem: Sri Lanka lacks a consolidated, machine-readable archive of public documents across law, governance, and media. Approach: the authors present Sri Lanka Document Datasets, a large-scale, multilingual repository built from 24 datasets in Sinhala, Tamil, and English, with automated crawling, parsing, and versioning. Contributions: open-source infrastructure under MIT license, daily updates, and mirrors on GitHub and Hugging Face, plus metadata standards and reproducible workflows. Significance: enables cross-lingual NLP, computational law, and policy analysis in a low-resource context while advancing open science and transparency.

Abstract

We present a collection of open, machine-readable document datasets covering parliamentary proceedings, legal judgments, government publications, news, and tourism statistics from Sri Lanka. The collection currently comprises of 230,091 documents (57.7 GB) across 24 datasets in Sinhala, Tamil, and English. The datasets are updated daily and mirrored on GitHub and Hugging Face. These resources aim to support research in computational linguistics, legal analytics, socio-political studies, and multilingual natural language processing. We describe the data sources, collection pipeline, formats, and potential use cases, while discussing licensing and ethical considerations. This manuscript is at version v2025-10-16-0818.