Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic

Sachin Goyal, Pratyush Maini, Zachary C. Lipton, Aditi Raghunathan, J. Zico Kolter

TL;DR

The paper introduces neural scaling laws for data filtering that explicitly account for the quality-quantity tradeoff in web data (QQT) and the compute budget, showing that high-quality data can lose utility with repetition while larger, lower-quality pools may yield better gains at scale. By modeling per-pool utility and decay, and deriving how mixtures of pools interact without training on their combinations, the authors provide a compute-aware framework for data curation and Pareto-frontier optimization. Empirical results on DataComp CLIP experiments reveal that aggressive filtering is optimal at low compute but suboptimal at high compute, while mixture-aware scaling curves can predict performance across diverse compute budgets and data pools. This work enables principled, compute-aware data curation strategies for large-scale visual-language models and highlights the need to rethink data filtering as a compute-constrained optimization problem.

Abstract

Vision-language models (VLMs) are trained for thousands of GPU hours on carefully curated web datasets. In recent times, data curation has gained prominence with several works developing strategies to retain 'high-quality' subsets of 'raw' scraped data. For instance, the LAION public dataset retained only 10% of the total crawled data. However, these strategies are typically developed agnostic of the available compute for training. In this paper, we first demonstrate that making filtering decisions independent of training compute is often suboptimal: the limited high-quality data rapidly loses its utility when repeated, eventually requiring the inclusion of 'unseen' but 'lower-quality' data. To address this quality-quantity tradeoff ($\texttt{QQT}$), we introduce neural scaling laws that account for the non-homogeneous nature of web data, an angle ignored in existing literature. Our scaling laws (i) characterize the $\textit{differing}$ 'utility' of various quality subsets of web data; (ii) account for how utility diminishes for a data point at its 'nth' repetition; and (iii) formulate the mutual interaction of various data pools when combined, enabling the estimation of model performance on a combination of multiple data pools without ever jointly training on them. Our key message is that data curation $\textit{cannot}$ be agnostic of the total compute that a model will be trained for. Our scaling laws allow us to curate the best possible pool for achieving top performance on Datacomp at various compute budgets, carving out a pareto-frontier for data curation. Code is available at https://github.com/locuslab/scaling_laws_data_filtering.

Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic

TL;DR

The paper introduces neural scaling laws for data filtering that explicitly account for the quality-quantity tradeoff in web data (QQT) and the compute budget, showing that high-quality data can lose utility with repetition while larger, lower-quality pools may yield better gains at scale. By modeling per-pool utility and decay, and deriving how mixtures of pools interact without training on their combinations, the authors provide a compute-aware framework for data curation and Pareto-frontier optimization. Empirical results on DataComp CLIP experiments reveal that aggressive filtering is optimal at low compute but suboptimal at high compute, while mixture-aware scaling curves can predict performance across diverse compute budgets and data pools. This work enables principled, compute-aware data curation strategies for large-scale visual-language models and highlights the need to rethink data filtering as a compute-constrained optimization problem.

Abstract

Vision-language models (VLMs) are trained for thousands of GPU hours on carefully curated web datasets. In recent times, data curation has gained prominence with several works developing strategies to retain 'high-quality' subsets of 'raw' scraped data. For instance, the LAION public dataset retained only 10% of the total crawled data. However, these strategies are typically developed agnostic of the available compute for training. In this paper, we first demonstrate that making filtering decisions independent of training compute is often suboptimal: the limited high-quality data rapidly loses its utility when repeated, eventually requiring the inclusion of 'unseen' but 'lower-quality' data. To address this quality-quantity tradeoff (), we introduce neural scaling laws that account for the non-homogeneous nature of web data, an angle ignored in existing literature. Our scaling laws (i) characterize the 'utility' of various quality subsets of web data; (ii) account for how utility diminishes for a data point at its 'nth' repetition; and (iii) formulate the mutual interaction of various data pools when combined, enabling the estimation of model performance on a combination of multiple data pools without ever jointly training on them. Our key message is that data curation be agnostic of the total compute that a model will be trained for. Our scaling laws allow us to curate the best possible pool for achieving top performance on Datacomp at various compute budgets, carving out a pareto-frontier for data curation. Code is available at https://github.com/locuslab/scaling_laws_data_filtering.
Paper Structure (61 sections, 1 theorem, 34 equations, 14 figures)

This paper contains 61 sections, 1 theorem, 34 equations, 14 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Data Filtering
  4. Scaling Laws in Language Modeling
  5. Scaling laws for downstream performance
  6. Scaling Laws in CLIP
  7. Data Filtering for a Compute Budget
  8. Experimental setup
  9. When "good" data performs worse
  10. Remark.
  11. Data filtering must be compute-aware
  12. Scaling Laws for Data Filtering
  13. Defining Utility
  14. Utility under Repetition
  15. Summary of Parameters
  16. ...and 46 more sections

Key Result

Theorem 1

Given $p$ data pools $\mathcal{S}_n^1$$\hdots$$\mathcal{S}_n^p$ sampled uniformly at random, with respective utility and repetition parameters given by $\left(b_1, \tau_1\right) \hdots \left(b_p, \tau_p\right)$, then the new repetition half-life of each of the buckets $\hat{\tau} = p\cdot \tau$. Add where $\hat{\delta}_i = \left(\frac{1}{2}\right)^{1/\hat{\tau}}$, the new decay parameter per bucke

Figures (14)

  • Figure 1: (a)The Dynamic Problem of Data Filtering: Web data is non-homogenous with subsets of varying quality (y-axis). For pretraining, "high-quality" data (such as bucket E) is limited in quantity and loses utility rapidly with repetitions (x-axis), which we call the quality-quantity tradeoff (QQT). Given a fixed compute budget (say equivalent to seeing 6 data pools), should we train on the best pool (E) for 6 epochs or on the 3 best pools (E, D, C) for 2 epochs each (in blue), and so on. How does the answer vary with the total compute budget? (b) We introduce scaling laws for data filtering that accommodate these new axes of heterogeneous and limited web data. We first model the (differing) initial utility, and rate of decay of utility (scaling parameters) of individual data pools (such as A--F in (a)). By deriving formulations for the mutual interaction of these buckets, we directly estimate the model performance when trained on combinations of these pools. Importantly, our methodology does not involve training on combinations of data pools even for estimating their scaling laws. Scatter points are true values for comparison, and the solid lines are extrapolated from the scaling parameters of individual buckets.
  • Figure 2: Given an initial data pool of 128M samples, we train ViT-B/32 CLIP models for a total of 640M samples. As we train for longer, the accuracy gains using the LAION-filtering subset that filters the common crawl to 10% of its initial size plateau. Surprisingly, even no-filtering of the common crawl is better than the popular LAION-filtering after seeing more than 450M samples.
  • Figure 3: We vary the CLIP filtering threshold after ranking the data by their metric. While the original paper proposed retaining 30% of the data, our results show that depending on the ratio of compute to data pool size, we must adaptively make the filtering less (or more) aggressive to account for the diminishing utility of good data with repetitions. Results are presented on an average of 18 visual understanding tasks with a global data pool size of 128M samples, and varying compute scales.
  • Figure 4: Scaling curves for various data quality pools: We partition the DataComp medium scale pool(128M) samples into various buckets, based on the T-MARS scores, and train a model on each bucket for upto 10 epochs. (a) The fitted error curves using the proposed scaling laws (Equation \ref{['eq:loss_effective_data']}). (b) Diminishing utilities with epochs of various data subsets. Observe that due to repetitions, even the utility of the best bucket (dark green) at it's $4^{\text{th}}$ repetition becomes lesser than that of worse buckets like top-20%-30% (orange curve) subset at it's $1^{\text{st}}$ epoch. This highlights why one needs to adapt the filtering aggressiveness with compute.
  • Figure 5: Estimating scaling curve for combination of various data pools : Using the estimated scaling parameters of individual data quality pools (Figure \ref{['fig:tmars_scaling_fit']}), we estimate the scaling law for the various combinations of the pools by modeling their mutual interaction (Theorem \ref{['thm:effective_utility']}). Note that we do not train on these data combinations to fit the above scaling curves (scatter points are test points), rather the scaling curves are estimated from the scaling parameters of individual pools. Scaling curves for average performance across 18tasks are in Figure \ref{['fig:tmars_18tasks_scaling_curve']}.
  • ...and 9 more figures

Theorems & Definitions (1)

  • Theorem 1