Towards Model-Agnostic Dataset Condensation by Heterogeneous Models

TL;DR

A novel method, Heterogeneous Model Dataset Condensation (HMDC), designed to produce universally applicable condensed images through cross-model interactions is introduced, and the Gradient Balance Module (GBM) and Mutual Distillation (MD) with the SpatialSemantic Decomposition method is proposed.

Abstract

Abstract. The advancement of deep learning has coincided with the proliferation of both models and available data. The surge in dataset sizes and the subsequent surge in computational requirements have led to the development of the Dataset Condensation (DC). While prior studies have delved into generating synthetic images through methods like distribution alignment and training trajectory tracking for more efficient model training, a significant challenge arises when employing these condensed images practically. Notably, these condensed images tend to be specific to particular models, constraining their versatility and practicality. In response to this limitation, we introduce a novel method, Heterogeneous Model Dataset Condensation (HMDC), designed to produce universally applicable condensed images through cross-model interactions. To address the issues of gradient magnitude difference and semantic distance in models when utilizing heterogeneous models, we propose the Gradient Balance Module (GBM) and Mutual Distillation (MD) with the SpatialSemantic Decomposition method. By balancing the contribution of each model and maintaining their semantic meaning closely, our approach overcomes the limitations associated with model-specific condensed images and enhances the broader utility. The source code is available in https://github.com/KHU-AGI/HMDC.

Figures (4)

• Figure 1: Accuracy plots illustrating the performance of different models trained using images generated by recent dataset condensation methods on the CIFAR-10 dataset with an IPC10 setting. Each bar signifies a performance comparison relative to randomly selected images on 10 images per class, with the initial state of each method identical to that of the random image. Notably, the methods exhibit over-condensation on ConvNet, resulting in performance degradation on other models.
• Figure 2: Diagram of Heterogeneous Model Dataset Condensation (HMDC), where two distinct models are employed for feature extraction. These features undergo dimension adjustment through Spatial-Semantic Decomposition, a critical step facilitating Mutual Distillation, and enhancing knowledge sharing between the two models. Throughout the dataset condensation process, the compensatory Gradient Balance Module comes into play, mitigating gradient variations inherent to different models. This module ensures the extraction of general knowledge by harmonizing gradient magnitudes, thus contributing to a more universally applicable condensation process.
• Figure 3: Comparision of condensed images between DREAM Liu_2023_ICCV and HMDC(Ours)
• Figure 4: A logarithmic plot depicting the gradient magnitude evolution of a synthetic image throughout the training process. L1 and L2 refer to the optimization targets in Eq. \ref{['eq:dual']}. L3 is $\mathrm{MSE}\left(\nabla\mathcal{L}_{\mathrm{MD}}(\textbf{x}^t),\nabla\mathcal{L}_{\mathrm{MD}}\right)$.