UDD: Dataset Distillation via Mining Underutilized Regions

TL;DR

UDD tackles the problem of underutilized regions in synthetic images produced by dataset distillation, which can limit performance when IPC is small. It introduces a utilization-sensitive framework with two policies—response-based and data jittering-based—to identify underutilized regions, and augments synthetic data by reusing these regions within a differentiable end-to-end pipeline. A Category-wise Feature Contrastive (CFC) loss complements gradient matching to enhance cross-class discrimination, with the overall objective $L_{syn} = L_g + L_c$ and a proxy network trained on real data to guide region utilization. Empirical results across MNIST, FashionMNIST, SVHN, CIFAR-10, and CIFAR-100 show state-of-the-art performance, particularly for IPC=1, along with a new mUE metric that correlates activation distribution with accuracy, underscoring improved data utilization and discriminability. The work offers practical gains for low-IPC condensation and provides a quantitative framework for measuring and optimizing synthetic data utilization.

Abstract

Dataset distillation synthesizes a small dataset such that a model trained on this set approximates the performance of the original dataset. Recent studies on dataset distillation focused primarily on the design of the optimization process, with methods such as gradient matching, feature alignment, and training trajectory matching. However, little attention has been given to the issue of underutilized regions in synthetic images. In this paper, we propose UDD, a novel approach to identify and exploit the underutilized regions to make them informative and discriminate, and thus improve the utilization of the synthetic dataset. Technically, UDD involves two underutilized regions searching policies for different conditions, i.e., response-based policy and data jittering-based policy. Compared with previous works, such two policies are utilization-sensitive, equipping with the ability to dynamically adjust the underutilized regions during the training process. Additionally, we analyze the current model optimization problem and design a category-wise feature contrastive loss, which can enhance the distinguishability of different categories and alleviate the shortcomings of the existing multi-formation methods. Experimentally, our method improves the utilization of the synthetic dataset and outperforms the state-of-the-art methods on various datasets, such as MNIST, FashionMNIST, SVHN, CIFAR-10, and CIFAR-100. For example, the improvements on CIFAR-10 and CIFAR-100 are 4.0\% and 3.7\% over the next best method with IPC=1, by mining the underutilized regions.

Figures (11)

• Figure 1: Examples of the synthetic images from 32 $\times$ 32 CIFAR-10 dataset. Left: synthetic images generated by DC DC, which has many underutilized regions (concentrated around the border). Right: synthetic images generated by IDC IDC, that the unbalanced information distribution is partially alleviated.
• Figure 2: Illustration of the proposed dataset distillation framework. Underutilized Region Searching: the module for mining the underutilized regions, which will be optimized with the original synthetic data in an end-to-end fashion. Candidate Regions: illustration of dividing each synthetic image into multiple regions. To facilitate observation, the regions in each sub-image are slightly staggered. Underutilized Regions: the underutilized regions will be resized to the original size by bilinear upsampling and combined with the original synthetic image to compose the synthetic training images.
• Figure 3: Examples of the synthetic images with their activation maps from 32 $\times$ 32 CIFAR-10 dataset. The synthetic images are generated by IDC IDC.
• Figure 4: Different schemes for addressing predefined regions (candidate regions) and their corresponding response. (a). Different predefined regions are built, and the network is run at all regions. (b). Different windows are run on the feature map.
• Figure 5: The data distribution of synthetic images from CIFAR-10, which trained on a randomly initialized neural network from scratch. The "numbers" with different colors represent the category corresponding to the number in CIFAR-10. For example, "0", "1", and "2" respectively represent "airplane", "automobile" and "bird".