Not All Instances Are Equally Valuable: Towards Influence-Weighted Dataset Distillation

TL;DR

Not all training instances contribute equally to dataset distillation; this work introduces Influence-Weighted Distillation (IWD), a modular framework that uses influence functions to assign adaptive weights to real data during distillation, emphasizing informative samples and downweighting harmful ones. By formulating a distillation objective that tracks matchable statistics along an inner training trajectory and computing per-instance influence scores, IWD reweights contributions during distillation and can be plugged into existing DD pipelines. Empirically, IWD improves distillation quality across benchmarks (CIFAR-10/100, SVHN) and architectures, achieving up to 7.8 percentage-point gains on CIFAR-10 and demonstrating robustness to hyperparameters and backbones. The approach offers a practical, principled way to leverage data quality in dataset condensation, with potential broad impact on storage, computation, and generalization in large-scale learning.

Abstract

Dataset distillation condenses large datasets into synthetic subsets, achieving performance comparable to training on the full dataset while substantially reducing storage and computation costs. Most existing dataset distillation methods assume that all real instances contribute equally to the process. In practice, real-world datasets contain both informative and redundant or even harmful instances, and directly distilling the full dataset without considering data quality can degrade model performance. In this work, we present Influence-Weighted Distillation IWD, a principled framework that leverages influence functions to explicitly account for data quality in the distillation process. IWD assigns adaptive weights to each instance based on its estimated impact on the distillation objective, prioritizing beneficial data while downweighting less useful or harmful ones. Owing to its modular design, IWD can be seamlessly integrated into diverse dataset distillation frameworks. Our empirical results suggest that integrating IWD tends to improve the quality of distilled datasets and enhance model performance, with accuracy gains of up to 7.8%.

Figures (5)

• Figure 1: Distribution of instance influence scores in dataset distillation. Most scores concentrate around zero, reflecting that most real instances contribute only marginally. Low-influence instances often come from redundant sub-categories (e.g., airliners), whereas high-influence instances, though rare (e.g., bombers, fighters, helicopters), provide valuable information for distillation.
• Figure 2: IWD Overview
• Figure 3: Ablation Study of Influence Weighting. Results on CIFAR10 ($IPC =1,10,50$) show that IWD consistently outperforms Random-Select, Influence-Select, and Influence-Prune, demonstrating the effectiveness of weighting instances by influence.
• Figure 4: Ablation studies on CIFAR10. (a) Ablation Study of $\tau$. Effect of softmax temperature $\tau$ under varying IPCs, showing a unimodal accuracy trend where moderate $\tau$ achieves the best balance between emphasizing high-influence instances and retaining global information. (b) Ablation Study of Architectures. Robustness across architectures (ConvNet-3, ResNet-10), where IWD consistently outperforms corresponding baselines, demonstrating generality beyond specific backbones.
• Figure 5: Synthesized images of CIFAR10 using IWD+ DC.