M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancy

TL;DR

Dataset condensation aims to reduce training data and cost while preserving performance. M3D advances DM-based condensation by embedding representations in an RKHS and minimizing the Maximum Mean Discrepancy, which captures infinite moments and aligns higher-order distribution properties. The approach combines theoretical guarantees (injective mean embeddings with universal kernels) with practical techniques (Factor & Up-sampling and Iteration per Random Model) to achieve SOTA results on ImageNet subsets and strong performance on smaller benchmarks, all with efficient training. This work provides a scalable, principled alternative to optimization-based condensation, offering both improved accuracy and broader applicability in realistic settings.

Abstract

Training state-of-the-art (SOTA) deep models often requires extensive data, resulting in substantial training and storage costs. To address these challenges, dataset condensation has been developed to learn a small synthetic set that preserves essential information from the original large-scale dataset. Nowadays, optimization-oriented methods have been the primary method in the field of dataset condensation for achieving SOTA results. However, the bi-level optimization process hinders the practical application of such methods to realistic and larger datasets. To enhance condensation efficiency, previous works proposed Distribution-Matching (DM) as an alternative, which significantly reduces the condensation cost. Nonetheless, current DM-based methods still yield less comparable results to SOTA optimization-oriented methods. In this paper, we argue that existing DM-based methods overlook the higher-order alignment of the distributions, which may lead to sub-optimal matching results. Inspired by this, we present a novel DM-based method named M3D for dataset condensation by Minimizing the Maximum Mean Discrepancy between feature representations of the synthetic and real images. By embedding their distributions in a reproducing kernel Hilbert space, we align all orders of moments of the distributions of real and synthetic images, resulting in a more generalized condensed set. Notably, our method even surpasses the SOTA optimization-oriented method IDC on the high-resolution ImageNet dataset. Extensive analysis is conducted to verify the effectiveness of the proposed method. Source codes are available at https://github.com/Hansong-Zhang/M3D.

Figures (11)

• Figure 1: Illustration of the importance of the higher-order alignment of distributions, where circles represent the representations of synthesized examples while crosses represent the representations of original examples. (a) The misaligned distributions with different second-order moments; (b) the misaligned distributions with different third-order moments; (c) the aligned distributions.
• Figure 2: The framework of M3D. After extracting the representations via a encoder network, the distributions of real and synthetic representations are further embedded in the Reproducing Kernel Hilbert Space (RKHS), where the M3D loss $\mathcal{L}_{\text{M3D}}$ is calculated to guild the update of synthetic examples for higher-order distribution alignment.
• Figure 3: Performance comparison between M3D and DM across varying training steps. M3D w/o Fac denotes the M3D without using the factor technique.
• Figure 4: Visualization of the condensed set of SVHN dataset with 10 images per class. The condensed set is generated by (b) DM and (c) M3D. Both DM and M3D use the same initialization as (a) shows.
• Figure 5: Representative samples condensed by M3D on ImageNet. The corresponding labels, from left to right and top to bottom, are bonnet, green snake, langur, doberman, gyromitra, saluki, vacuum, window screen, and cockroach.

Theorems & Definitions (3)

• Definition 1
• Theorem 1
• Theorem 2