Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Position: The Most Expensive Part of an LLM should be its Training Data

Nikhil Kandpal, Colin Raffel

TL;DR

The paper argues that the labor behind training data is the most expensive input in LLM development and remains largely uncompensated, with data costs often dwarfing compute costs. It adopts a replacement-cost, labor-based valuation approach to quantify data labor and compares it to traditional training costs using a prior methodology. Findings show dataset costs exceed training costs by large margins and are projected to grow with continued scaling, posing affordability and fairness challenges. The authors propose research directions—data collection reforms, data-efficient training, and flexible compensation models—to enable fairer value distribution and more sustainable AI development.

Abstract

Training a state-of-the-art Large Language Model (LLM) is an increasingly expensive endeavor due to growing computational, hardware, energy, and engineering demands. Yet, an often-overlooked (and seldom paid) expense is the human labor behind these models' training data. Every LLM is built on an unfathomable amount of human effort: trillions of carefully written words sourced from books, academic papers, codebases, social media, and more. This position paper aims to assign a monetary value to this labor and argues that the most expensive part of producing an LLM should be the compensation provided to training data producers for their work. To support this position, we study 64 LLMs released between 2016 and 2024, estimating what it would cost to pay people to produce their training datasets from scratch. Even under highly conservative estimates of wage rates, the costs of these models' training datasets are 10-1000 times larger than the costs to train the models themselves, representing a significant financial liability for LLM providers. In the face of the massive gap between the value of training data and the lack of compensation for its creation, we highlight and discuss research directions that could enable fairer practices in the future.

Position: The Most Expensive Part of an LLM should be its Training Data

TL;DR

The paper argues that the labor behind training data is the most expensive input in LLM development and remains largely uncompensated, with data costs often dwarfing compute costs. It adopts a replacement-cost, labor-based valuation approach to quantify data labor and compares it to traditional training costs using a prior methodology. Findings show dataset costs exceed training costs by large margins and are projected to grow with continued scaling, posing affordability and fairness challenges. The authors propose research directions—data collection reforms, data-efficient training, and flexible compensation models—to enable fairer value distribution and more sustainable AI development.

Abstract

Training a state-of-the-art Large Language Model (LLM) is an increasingly expensive endeavor due to growing computational, hardware, energy, and engineering demands. Yet, an often-overlooked (and seldom paid) expense is the human labor behind these models' training data. Every LLM is built on an unfathomable amount of human effort: trillions of carefully written words sourced from books, academic papers, codebases, social media, and more. This position paper aims to assign a monetary value to this labor and argues that the most expensive part of producing an LLM should be the compensation provided to training data producers for their work. To support this position, we study 64 LLMs released between 2016 and 2024, estimating what it would cost to pay people to produce their training datasets from scratch. Even under highly conservative estimates of wage rates, the costs of these models' training datasets are 10-1000 times larger than the costs to train the models themselves, representing a significant financial liability for LLM providers. In the face of the massive gap between the value of training data and the lack of compensation for its creation, we highlight and discuss research directions that could enable fairer practices in the future.

Paper Structure

This paper contains 21 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Estimating LLM Development Costs
  3. Estimating Training Costs
  4. Estimating Dataset Production Costs
  5. Cost Analysis Results
  6. LLM Dataset Costs Outweigh Training Costs
  7. Few Companies Can Afford Large-Scale Datasets
  8. Training Dataset Costs Will Continue to Grow
  9. Future Research Directions
  10. Training Data Collection
  11. Permissively Licensed and Public Domain Text
  12. Opt-In Data Collection
  13. Authorship Provenance
  14. Balancing Compute and Data Costs
  15. Data Efficiency
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Estimated costs for LLMs' training datasets surpass the costs of training by 1-3 orders of magnitude. Above, we plot the training costs of 64 language models released between 2016 and 2024 along with the estimated cost of their training datasets.
  • Figure 2: Over time the estimated labor costs to produce the content of LLM training datasets has increased, with numerous recent models having been trained on datasets that we conservatively estimate to have implicitly cost over $10 billion USD.
  • Figure 3: The costs of training datasets used by major LLM companies make up a significant fraction of these companies' revenues. Above we visualize training data cost as a percentage of each company's annual revenue for ten recent LLM training runs. We denote non-publicly traded companies denoted with * to indicate that for these companies we use third-party reported revenue rather than revenues reported in official financial filings.