Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Utility Boundary of Dataset Distillation: Scaling and Configuration-Coverage Laws

Zhengquan Luo, Zhiqiang Xu

TL;DR

This work develops a unified configuration–dynamics–error framework to ground dataset distillation (DD) in theory, showing that gradient-, distribution-, and trajectory-matching methods are instances of a single alignment objective. It proves a single-configuration scaling law where generalization error decays as $O(1/\sqrt{k})$ up to an irreducible floor, and a coverage law stating that the required distilled size grows linearly with configuration diversity. The authors validate these laws across multiple DD methods and datasets, providing a principled explanation for IPC saturation and robustness under training-configuration shifts. The results offer theory-driven guidance for designing compact, configuration-robust synthetic datasets with provable guarantees.

Abstract

Dataset distillation (DD) aims to construct compact synthetic datasets that allow models to achieve comparable performance to full-data training while substantially reducing storage and computation. Despite rapid empirical progress, its theoretical foundations remain limited: existing methods (gradient, distribution, trajectory matching) are built on heterogeneous surrogate objectives and optimization assumptions, which makes it difficult to analyze their common principles or provide general guarantees. Moreover, it is still unclear under what conditions distilled data can retain the effectiveness of full datasets when the training configuration, such as optimizer, architecture, or augmentation, changes. To answer these questions, we propose a unified theoretical framework, termed configuration--dynamics--error analysis, which reformulates major DD approaches under a common generalization-error perspective and provides two main results: (i) a scaling law that provides a single-configuration upper bound, characterizing how the error decreases as the distilled sample size increases and explaining the commonly observed performance saturation effect; and (ii) a coverage law showing that the required distilled sample size scales linearly with configuration diversity, with provably matching upper and lower bounds. In addition, our unified analysis reveals that various matching methods are interchangeable surrogates, reducing the same generalization error, clarifying why they can all achieve dataset distillation and providing guidance on how surrogate choices affect sample efficiency and robustness. Experiments across diverse methods and configurations empirically confirm the derived laws, advancing a theoretical foundation for DD and enabling theory-driven design of compact, configuration-robust dataset distillation.

Utility Boundary of Dataset Distillation: Scaling and Configuration-Coverage Laws

TL;DR

This work develops a unified configuration–dynamics–error framework to ground dataset distillation (DD) in theory, showing that gradient-, distribution-, and trajectory-matching methods are instances of a single alignment objective. It proves a single-configuration scaling law where generalization error decays as up to an irreducible floor, and a coverage law stating that the required distilled size grows linearly with configuration diversity. The authors validate these laws across multiple DD methods and datasets, providing a principled explanation for IPC saturation and robustness under training-configuration shifts. The results offer theory-driven guidance for designing compact, configuration-robust synthetic datasets with provable guarantees.

Abstract

Dataset distillation (DD) aims to construct compact synthetic datasets that allow models to achieve comparable performance to full-data training while substantially reducing storage and computation. Despite rapid empirical progress, its theoretical foundations remain limited: existing methods (gradient, distribution, trajectory matching) are built on heterogeneous surrogate objectives and optimization assumptions, which makes it difficult to analyze their common principles or provide general guarantees. Moreover, it is still unclear under what conditions distilled data can retain the effectiveness of full datasets when the training configuration, such as optimizer, architecture, or augmentation, changes. To answer these questions, we propose a unified theoretical framework, termed configuration--dynamics--error analysis, which reformulates major DD approaches under a common generalization-error perspective and provides two main results: (i) a scaling law that provides a single-configuration upper bound, characterizing how the error decreases as the distilled sample size increases and explaining the commonly observed performance saturation effect; and (ii) a coverage law showing that the required distilled sample size scales linearly with configuration diversity, with provably matching upper and lower bounds. In addition, our unified analysis reveals that various matching methods are interchangeable surrogates, reducing the same generalization error, clarifying why they can all achieve dataset distillation and providing guidance on how surrogate choices affect sample efficiency and robustness. Experiments across diverse methods and configurations empirically confirm the derived laws, advancing a theoretical foundation for DD and enabling theory-driven design of compact, configuration-robust dataset distillation.

Paper Structure

This paper contains 76 sections, 24 theorems, 323 equations, 3 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries and Problem Setup
  3. A Unified configuration-Dynamics-Error Framework
  4. Single–configuration Generalization Bound
  5. Coverage-Aware Bounds: Risk Across Training Configurations
  6. Unifying various categories of dataset distillation
  7. Experiments
  8. Conclusion and Limitations
  9. Notations
  10. Related work
  11. Methodological Advances in Dataset Distillation
  12. Theoretical Explorations in Dataset Distillation
  13. Proofs for the Single--configuration Bound
  14. Technical lemmas used in the proofs
  15. Step C.5.2 Symmetrization.
  16. ...and 61 more sections

Key Result

Theorem 4.2

Let $\theta_T^{(s)}$ and $\theta_T^{(\tau)}$ denote the parameters after $T$ steps trained on synthetic and real data, respectively, with initialization gap $\delta_0=\theta_0^{(s)}-\theta_0^{(\tau)}$. Then with probability at least $1-\varepsilon$, where $e_g=\mathcal{O}(1/\sqrt{k}+1/\sqrt{n})$ is the fluctuation from distillation and training samples (see, e.g., bartlett2002rademacher, for Rade

Figures (3)

  • Figure 1: Configuration-dynamics-error framework: a configuration (optimizer, architecture, augmentation, etc.) together with the training distribution (either the real dataset or its distilled dataset) induces optimization dynamics, whose risk is evaluated through generalization error bounds; this yields the scaling law for a single configuration and the coverage law across configuration families.
  • Figure 2: Single-configuration scaling law. On MNIST, CIFAR-10/100, and ImageNette, the curves of generalization error $\Delta$ against $1/\sqrt{k}$ for GM, DM, and MTT shows linear decay at small $k$ followed by saturation at a positive generalization error bound. Regression intercepts give $\epsilon_{\mathrm{bound}}(a)$, consistent with Theorems \ref{['thm:single']} and \ref{['thm:dynamic-single']}.
  • Figure 3: Configuration coverage law. For subsets of $m$ configurations, plotting $Y=\Delta(k,M)$ against $X=\sqrt{\log M}/\sqrt{k}$ yields near-linear trends under random sampling. Results remain consistent with Theorems \ref{['thm:uniform-cross-configuration']} and \ref{['thm:dynamic-coverage']}.

Theorems & Definitions (52)

  • Definition 4.1: Intrinsic generalization error
  • Theorem 4.2: Single–configuration risk bound
  • Remark 4.3
  • Corollary 4.4
  • Remark 4.5
  • Theorem 5.1: Uniform cross--configuration bound
  • Theorem 5.2: Coverage lower bound
  • Corollary 5.3: Coverage law
  • Remark 5.4: Coverage law
  • Remark 5.5: Practical estimation of coverage entropy
  • ...and 42 more