Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data Valuation for LLM Fine-Tuning: Efficient Shapley Value Approximation via Language Model Arithmetic

Mélissa Tamine, Otmane Sakhi, Benjamin Heymann

TL;DR

This paper tackles data valuation for LLM fine-tuning by leveraging Direct Preference Optimization (DPO) and language model arithmetic to approximate Shapley values with a linear number of fine-tunings rather than an exponential number. By training one model per data source and composing these models at inference, coalition utilities can be obtained without coalition-specific training, enabling efficient Shapley estimation. The approach enables multi-objective analysis (e.g., helpfulness vs. harmlessness) and suggests broader uses in data curation, weighting, and dynamic training architectures. Overall, the work provides a practical path toward fair and scalable data valuation for LLMs, with potential implications for data markets, ownership, and responsible training practices.

Abstract

Data is a critical asset for training large language models (LLMs), alongside compute resources and skilled workers. While some training data is publicly available, substantial investment is required to generate proprietary datasets, such as human preference annotations or to curate new ones from existing sources. As larger datasets generally yield better model performance, two natural questions arise. First, how can data owners make informed decisions about curation strategies and data sources investment? Second, how can multiple data owners collaboratively pool their resources to train superior models while fairly distributing the benefits? This problem, data valuation, which is not specific to large language models, has been addressed by the machine learning community through the lens of cooperative game theory, with the Shapley value being the prevalent solution concept. However, computing Shapley values is notoriously expensive for data valuation, typically requiring numerous model retrainings, which can become prohibitive for large machine learning models. In this work, we demonstrate that this computational challenge is dramatically simplified for LLMs trained with Direct Preference Optimization (DPO). We show how the specific mathematical structure of DPO enables scalable Shapley value computation. We believe this observation unlocks many applications at the intersection of data valuation and large language models.

Data Valuation for LLM Fine-Tuning: Efficient Shapley Value Approximation via Language Model Arithmetic

TL;DR

This paper tackles data valuation for LLM fine-tuning by leveraging Direct Preference Optimization (DPO) and language model arithmetic to approximate Shapley values with a linear number of fine-tunings rather than an exponential number. By training one model per data source and composing these models at inference, coalition utilities can be obtained without coalition-specific training, enabling efficient Shapley estimation. The approach enables multi-objective analysis (e.g., helpfulness vs. harmlessness) and suggests broader uses in data curation, weighting, and dynamic training architectures. Overall, the work provides a practical path toward fair and scalable data valuation for LLMs, with potential implications for data markets, ownership, and responsible training practices.

Abstract

Data is a critical asset for training large language models (LLMs), alongside compute resources and skilled workers. While some training data is publicly available, substantial investment is required to generate proprietary datasets, such as human preference annotations or to curate new ones from existing sources. As larger datasets generally yield better model performance, two natural questions arise. First, how can data owners make informed decisions about curation strategies and data sources investment? Second, how can multiple data owners collaboratively pool their resources to train superior models while fairly distributing the benefits? This problem, data valuation, which is not specific to large language models, has been addressed by the machine learning community through the lens of cooperative game theory, with the Shapley value being the prevalent solution concept. However, computing Shapley values is notoriously expensive for data valuation, typically requiring numerous model retrainings, which can become prohibitive for large machine learning models. In this work, we demonstrate that this computational challenge is dramatically simplified for LLMs trained with Direct Preference Optimization (DPO). We show how the specific mathematical structure of DPO enables scalable Shapley value computation. We believe this observation unlocks many applications at the intersection of data valuation and large language models.

Paper Structure

This paper contains 13 sections, 13 equations, 2 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related works.
  3. Preliminary
  4. Evaluation of LLMs
  5. Shapley-based data valuation
  6. LLM fine-tuning
  7. Coalition policies and Language model arithmetic
  8. Implementation
  9. Remark.
  10. Vision
  11. The Shapley value, and more generally semivalues, could be used to quantify, in an interpretable manner, the contribution of each dataset used during training
  12. Since data quality is key to LLM performance, one could envision new training algorithms that automatically curate or weight their input data, just as it is now typical in machine learning to reweight datasets or individual datapoints to improve model outcomes.
  13. When datasets reflect individual preferences

Figures (2)

  • Figure 1: We can equip the collections of datasets, the set of reward models, and the set of policies with binary operators to give them a group (or semi-group) structure. After doing so, we observe that some training methods, such as sequential DPO, preserve this structure and therefore induce morphisms between those semi-groups.
  • Figure 2: Approximate Shapley values of $4$ UltraFeedback data sources (flan_v2_niv2, sharegpt, evol_instruct, ultrachat) under two rewards. Each point corresponds to a data source. The x-axis shows its Shapley value for the helpfulness reward, and the y-axis for the harmlessness reward. The diagonal $y = x$ indicates perfect agreement between the two rewards on the relative importance of each source.