Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fairshare Data Pricing via Data Valuation for Large Language Models

Luyang Zhang, Cathy Jiao, Beibei Li, Chenyan Xiong

TL;DR

This work addresses fair data pricing for LLMs by framing the data market as a Stackelberg game where buyers and sellers exchange datasets valued by their contribution to model performance. It introduces fairshare pricing, derived from data valuation signals, to sustain seller participation and maximize long-term buyer utility, contrasting it with exploitative pricing that destabilizes the data supply. Theoretical analyses show exploitative pricing leads to market collapse, while fairshare pricing yields a mutually beneficial equilibrium. Empirical simulations across NLP tasks demonstrate higher seller profits, stable data supply, and improved buyers’ cost-efficiency, with robustness to various valuation methods. The approach offers a practical blueprint for transparent, equitable, and economically sustainable LLM data markets, with implications for policy and platform design.

Abstract

Training data is the backbone of large language models (LLMs), yet today's data markets often operate under exploitative pricing -- sourcing data from marginalized groups with little pay or recognition. This paper introduces a theoretical framework for LLM data markets, modeling the strategic interactions between buyers (LLM builders) and sellers (human annotators). We begin with theoretical and empirical analysis showing how exploitative pricing drives high-quality sellers out of the market, degrading data quality and long-term model performance. Then we introduce fairshare, a pricing mechanism grounded in data valuation that quantifies each data's contribution. It aligns incentives by sustaining seller participation and optimizing utility for both buyers and sellers. Theoretically, we show that fairshare yields mutually optimal outcomes: maximizing long-term buyer utility and seller profit while sustaining market participation. Empirically when training open-source LLMs on complex NLP tasks, including math problems, medical diagnosis, and physical reasoning, fairshare boosts seller earnings and ensures a stable supply of high-quality data, while improving buyers' performance-per-dollar and long-term welfare. Our findings offer a concrete path toward fair, transparent, and economically sustainable data markets for LLM.

Fairshare Data Pricing via Data Valuation for Large Language Models

TL;DR

This work addresses fair data pricing for LLMs by framing the data market as a Stackelberg game where buyers and sellers exchange datasets valued by their contribution to model performance. It introduces fairshare pricing, derived from data valuation signals, to sustain seller participation and maximize long-term buyer utility, contrasting it with exploitative pricing that destabilizes the data supply. Theoretical analyses show exploitative pricing leads to market collapse, while fairshare pricing yields a mutually beneficial equilibrium. Empirical simulations across NLP tasks demonstrate higher seller profits, stable data supply, and improved buyers’ cost-efficiency, with robustness to various valuation methods. The approach offers a practical blueprint for transparent, equitable, and economically sustainable LLM data markets, with implications for policy and platform design.

Abstract

Training data is the backbone of large language models (LLMs), yet today's data markets often operate under exploitative pricing -- sourcing data from marginalized groups with little pay or recognition. This paper introduces a theoretical framework for LLM data markets, modeling the strategic interactions between buyers (LLM builders) and sellers (human annotators). We begin with theoretical and empirical analysis showing how exploitative pricing drives high-quality sellers out of the market, degrading data quality and long-term model performance. Then we introduce fairshare, a pricing mechanism grounded in data valuation that quantifies each data's contribution. It aligns incentives by sustaining seller participation and optimizing utility for both buyers and sellers. Theoretically, we show that fairshare yields mutually optimal outcomes: maximizing long-term buyer utility and seller profit while sustaining market participation. Empirically when training open-source LLMs on complex NLP tasks, including math problems, medical diagnosis, and physical reasoning, fairshare boosts seller earnings and ensures a stable supply of high-quality data, while improving buyers' performance-per-dollar and long-term welfare. Our findings offer a concrete path toward fair, transparent, and economically sustainable data markets for LLM.

Paper Structure

This paper contains 31 sections, 7 theorems, 42 equations, 14 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. A Theoretical Framework of LLM Data Market
  4. Data Market Definition
  5. Exploitative Pricing: A Lose-Lose Outcome
  6. Fairshare Pricing: A Win-Win for Sellers and Buyers
  7. Empirical Analyses: Benefits of Fairshare Pricing
  8. Experimental Setup
  9. LLM Data Market Experiments
  10. Ablation Study: Effect of Different Data Valuation Methods
  11. Conclusion
  12. Influence-based Data Valuation
  13. Royalty model
  14. Solving for optimal price of the royalty model
  15. Applications for Real-Life Scenarios
  16. ...and 16 more sections

Key Result

Lemma 1

With ass:seller_participationass:prob_functionass:discount_factor, any exploitative pricing (i.e., $p_t < p_{t}^*, \forall t$) will only maximize cumulative utility within a finite horizon -- after which it is strictly suboptimal.

Figures (14)

  • Figure 1: Illustration of the LLM data market, showing buyers (LLM builders) and sellers (annotators) interacting via a pricing mechanism based on game theory and data valuation.
  • Figure 2: Buyer's cumulative utility and seller participation under ideal and exploitative pricing (\ref{['fig:explo_pricing_1']},\ref{['fig:explo_pricing_2']}); Profits of sellers $S_{1}$ and $S_{2}$ with buyer $B_{1}$'s and $B_{2}$'s MWP over price (\ref{['fig:demo_opt_sol_1']},\ref{['fig:demo_opt_sol_2']}).
  • Figure 3: Analysis of the buyer's cumulative net utility as a function of the acquisition prices over time ($T = 1, 2, 5, 10$). Note: the market has one buyer and seller.
  • Figure 4: (1) buyer's cumulative utilities with high- (\ref{['fig:exp_pricing_buyer_medqa_pythia_high']}) and low-budget buyer (\ref{['fig:exp_pricing_buyer_medqa_pythia_low']}), and (2) sellers' average cumulative profits (\ref{['fig:exp_pricing_seller_medqa_pythia_profit']}) and active seller numbers (\ref{['fig:exp_pricing_seller_medqa_pythia_num']}) over $100$ time periods. Pythia-1b; MedQA; Groups: (1) fairshare, (2) reduced, (3) random, and (4) exploitative.
  • Figure 5: Purchased datasets for the buyer with high budget (\ref{['fig:exp_pricing_Analysis_mechansim_pythia_high']}) and low budget (\ref{['fig:exp_pricing_Analysis_mechansim_pythia_low']}) over $100$ time periods. Pythia-1b; MedQA.
  • ...and 9 more figures

Theorems & Definitions (15)

  • Lemma 1: Inevitable Failure of Exploitative Pricing
  • Lemma 2: Characterization of Optimal Price $p_{j}^*$
  • Lemma 3: The Optimal Price for Buyer Is Also $p^*_{t}$
  • Lemma 4: The Trade-Off Threshold Is Increasing as $\delta$ Decreases
  • Lemma 5
  • Lemma 6: Characterization of $\alpha_{j}^*$ under royalty model
  • Remark B.1: Similarities between flat rate and royalty model
  • proof
  • Lemma 7: Participation Loss Is Lower-Bounded
  • proof
  • ...and 5 more