The KL3M Data Project: Copyright-Clean Training Resources for Large Language Models

TL;DR

The KL3M Data Project tackles the legal ambiguity surrounding training data for large language models by instituting a formal, three-test data protocol that screens sources for copyright status and licensing terms. It delivers a scalable, provenance-rich pipeline that converts original documents into standardized, training-ready representations while preserving source provenance and offering mid- and post-train resources. The work provides a legally grounded alternative to traditional, copyright-heavy corpora, with an extensive, license-verified collection of 132 million documents and 28 TB of data across government, regulatory, and domain-specific sources. By releasing open-source tooling, data, and metadata under permissive licenses, KL3M aims to enable ethical, sustainable AI development and invites collaboration to broaden jurisdictional and domain coverage along with robust attribution and currency mechanisms.

Abstract

Practically all large language models have been pre-trained on data that is subject to global uncertainty related to copyright infringement and breach of contract. This creates potential risk for users and developers due to this uncertain legal status. The KL3M Data Project directly confronts this critical issue by introducing the largest comprehensive training data pipeline that minimizes risks related to copyright or breach of contract. The foundation of this project is a corpus of over 132 million documents and trillions of tokens spanning 16 different sources that have been verified to meet the strict copyright and licensing protocol detailed herein. We are releasing the entire pipeline, including 1) the source code to acquire and process these documents, 2) the original document formats with associated provenance and metadata, 3) extracted content in a standardized format, 4) pre-tokenized representations of the documents, and 5) various mid- and post-train resources such as question-answer, summarization, conversion, drafting, classification, prediction, and conversational data. All of these resources are freely available to the public on S3, Hugging Face, and GitHub under CC-BY terms. We are committed to continuing this project in furtherance of a more ethical, legal, and sustainable approach to the development and use of AI models.

Figures (7)

• Figure 1: The KL3M Data Protocol determines whether to include content through three sequential tests that address both copyright and contract risks identified in Section \ref{['sec:legal_problem_space']}.
• Figure 2: Three-stage data architecture. Documents progress from their original formats with complete metadata to standardized text representations, and finally to storage-optimized formats for model training. Each stage maintains references to previous stages, enabling provenance tracking and future re-processing.
• Figure 3: The "Northeastern Loggers' Handbook" from the Federal Depository Library Program (FDLP) showing the original document preserved in our collection. This document and its associated metadata can be viewed https://gallery.kl3m.ai/document/view?identifier=fdlp/gpo16926/PDF.pdf.json.
• Figure 4: Document counts and storage size across the three stages in s3://data.kl3m.ai/
• Figure 5: Sample of four documents that illustrate the subject matter diversity. View more at https://gallery.kl3m.ai/