Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Grounding and Enhancing Informativeness and Utility in Dataset Distillation

Shaobo Wang, Yantai Yang, Guo Chen, Peiru Li, Kaixin Li, Yufa Zhou, Zhaorun Chen, Linfeng Zhang

TL;DR

The paper formalizes dataset distillation through Informativeness and Utility, enabling a principled optimal distillation objective. It introduces InfoUtil, a two-step framework that uses Shapley-value attributions to extract informative patches and gradient-norm scores to preserve high-utility samples, yielding interpretable and effective distilled datasets. Empirically, InfoUtil achieves state-of-the-art gains on ImageNet-scale tasks (e.g., +16% on ImageNet-100 and +6.1% on ImageNet-1K) with substantial efficiency advantages over prior methods. This approach advances the practicality and interpretability of dataset condensation, offering a scalable pathway for high-quality synthetic data in vision applications.

Abstract

Dataset Distillation (DD) seeks to create a compact dataset from a large, real-world dataset. While recent methods often rely on heuristic approaches to balance efficiency and quality, the fundamental relationship between original and synthetic data remains underexplored. This paper revisits knowledge distillation-based dataset distillation within a solid theoretical framework. We introduce the concepts of Informativeness and Utility, capturing crucial information within a sample and essential samples in the training set, respectively. Building on these principles, we define optimal dataset distillation mathematically. We then present InfoUtil, a framework that balances informativeness and utility in synthesizing the distilled dataset. InfoUtil incorporates two key components: (1) game-theoretic informativeness maximization using Shapley Value attribution to extract key information from samples, and (2) principled utility maximization by selecting globally influential samples based on Gradient Norm. These components ensure that the distilled dataset is both informative and utility-optimized. Experiments demonstrate that our method achieves a 6.1\% performance improvement over the previous state-of-the-art approach on ImageNet-1K dataset using ResNet-18.

Grounding and Enhancing Informativeness and Utility in Dataset Distillation

TL;DR

The paper formalizes dataset distillation through Informativeness and Utility, enabling a principled optimal distillation objective. It introduces InfoUtil, a two-step framework that uses Shapley-value attributions to extract informative patches and gradient-norm scores to preserve high-utility samples, yielding interpretable and effective distilled datasets. Empirically, InfoUtil achieves state-of-the-art gains on ImageNet-scale tasks (e.g., +16% on ImageNet-100 and +6.1% on ImageNet-1K) with substantial efficiency advantages over prior methods. This approach advances the practicality and interpretability of dataset condensation, offering a scalable pathway for high-quality synthetic data in vision applications.

Abstract

Dataset Distillation (DD) seeks to create a compact dataset from a large, real-world dataset. While recent methods often rely on heuristic approaches to balance efficiency and quality, the fundamental relationship between original and synthetic data remains underexplored. This paper revisits knowledge distillation-based dataset distillation within a solid theoretical framework. We introduce the concepts of Informativeness and Utility, capturing crucial information within a sample and essential samples in the training set, respectively. Building on these principles, we define optimal dataset distillation mathematically. We then present InfoUtil, a framework that balances informativeness and utility in synthesizing the distilled dataset. InfoUtil incorporates two key components: (1) game-theoretic informativeness maximization using Shapley Value attribution to extract key information from samples, and (2) principled utility maximization by selecting globally influential samples based on Gradient Norm. These components ensure that the distilled dataset is both informative and utility-optimized. Experiments demonstrate that our method achieves a 6.1\% performance improvement over the previous state-of-the-art approach on ImageNet-1K dataset using ResNet-18.
Paper Structure (33 sections, 1 theorem, 29 equations, 9 figures, 11 tables, 1 algorithm)

This paper contains 33 sections, 1 theorem, 29 equations, 9 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Challenge 1: Efficiency-Performance Trade-off.
  3. Challenge 2: Lack of Interpretability.
  4. Preliminaries
  5. Method
  6. Optimal Dataset Distillation
  7. InfoUtil
  8. Game-theoretic Informativeness Maximization
  9. Principled Utility Maximization
  10. Experiments
  11. Experimental Settings
  12. Main Results
  13. Ablation studies
  14. Discussion
  15. Conclusion
  16. ...and 18 more sections

Key Result

Theorem 1

Let the utility function $\mathcal{U}$ be defined as in Definition def:utility. Then there exists a constant $c > 0$ such that

Figures (9)

  • Figure 1: Comparison of visualization results between previous method (a) RDED RDED and (b) our InfoUtil. Unlike prior methods relying on random selection and intuitive scoring, InfoUtil is both interpretable and theoretically grounded. It synthesizes images that more accurately capture semantically meaningful regions with principled scores. Prioritizing core content over irrelevant details like background elements ensures a more focused and meaningful representation.
  • Figure 2: InfoUtil's pipeline for optimal dataset distillation involves two key steps: (i) Step 1 maximizes informativeness via the Shapley Value (a game-theoretic attribution method), retaining the most informative patches to form compressed samples. (ii) Step 2 maximizes utility by scoring these candidates with a judge model—using Gradient Norm (proven as a utility upper bound)—and retaining top samples. The final distilled dataset contains only the most informative, high-utility compressed samples. Image reconstruction and soft label generation phases are omitted here.
  • Figure 3: Performance comparison on ResNet-18 and MobileNet. (a) Time cost in seconds (lower is better): "TB" denotes training-based methods (TESLA and SRe2L fall into this category); "TF" denotes training-free methods (others belong to this type). (b) Peak memory in GB (lower is better): InfoUtil performs competitively with far lower costs than training-based methods. "Info" denotes Informativeness only, while "Util" denotes Utility only.
  • Figure 4: Analysis of teacher networks for soft label generation. ConvNet performance on ImageWoof using labels from five training stages (IPC=1/10). "Full" denotes pretrained teacher. (a) IPC=1: Early high-entropy labels beat full model, aiding low-data scenarios. (b) IPC=10: Full model's low-entropy labels excel in data-rich conditions.
  • Figure 5: Visualization of condensed images for the indigo bunting category on ImageNet-1K.
  • ...and 4 more figures

Theorems & Definitions (6)

  • Definition 1: Informativeness
  • Definition 2: Gradient Flow
  • Definition 3: Utility
  • Definition 4: Optimal Dataset Distillation
  • Definition 5: Gradient Norm
  • Theorem 1: Utility is bounded by Gradient Norm. Proof in Appendix \ref{['appendix:proof']}