DANCE: Dual-View Distribution Alignment for Dataset Condensation

TL;DR

DANCE addresses the efficiency-accuracy gap in dataset condensation by augmenting Distribution Matching with two expert-guided views. The inner-class view employs Pseudo Long-Term Distribution Alignment (PLTDA) using middle encoders to maintain faithful class representations throughout training, while the inter-class view uses Distribution Calibration with expert models to keep synthetic data within real class regions. Empirically, DANCE achieves state-of-the-art results across a range of datasets and resolutions, with notable cross-architecture generalization and improved condensation efficiency. This dual-view framework significantly reduces data burdens for practical, large-scale model development while maintaining competitive performance.

Abstract

Dataset condensation addresses the problem of data burden by learning a small synthetic training set that preserves essential knowledge from the larger real training set. To date, the state-of-the-art (SOTA) results are often yielded by optimization-oriented methods, but their inefficiency hinders their application to realistic datasets. On the other hand, the Distribution-Matching (DM) methods show remarkable efficiency but sub-optimal results compared to optimization-oriented methods. In this paper, we reveal the limitations of current DM-based methods from the inner-class and inter-class views, i.e., Persistent Training and Distribution Shift. To address these problems, we propose a new DM-based method named Dual-view distribution AligNment for dataset CondEnsation (DANCE), which exploits a few pre-trained models to improve DM from both inner-class and inter-class views. Specifically, from the inner-class view, we construct multiple "middle encoders" to perform pseudo long-term distribution alignment, making the condensed set a good proxy of the real one during the whole training process; while from the inter-class view, we use the expert models to perform distribution calibration, ensuring the synthetic data remains in the real class region during condensing. Experiments demonstrate the proposed method achieves a SOTA performance while maintaining comparable efficiency with the original DM across various scenarios. Source codes are available at https://github.com/Hansong-Zhang/DANCE.

Figures (15)

• Figure 1: Two views of the proposed DANCE. For inner-class view, it ensures that the synthetic data remains a faithful proxy of the real data throughout the training process. For inter-class view, it also prevents the synthetic data from falling outside the real class region (the domain where all real data points of that class reside), which may change the decision boundary of the learned classifier.
• Figure 2: The framework of DANCE. From the inner-class view, multiple middle encoders are constructed to perform Pseudo Long-Term Distribution Alignment so that the synthetic set can remain a good proxy of its class during training. From the inter-class view, the Distribution Calibration is performed, ensuring the synthetic data stay within the real class region during condensing process.
• Figure 3: (a) The discrepancy between the feature distribution of $\mathcal{D}_\mathsf{real}$ and $\mathcal{D}_\mathsf{syn}$ of DM and DANCE across the whole training process with the real training data. (b) The test accuracy(%) of the middle model $\bm{\phi}_\mathsf{mid}$ at different value of $\lambda$. (c) The accuracy (%) of the expert model on the real test data, and the synthesized data of DM and DANCE. The evaluations are conducted on CIFAR-10, where (a) and (c) adopts 10 images per class.
• Figure 4: Example condensed images of $32\times 32$CIFAR-10, $128\times 128$ImageSquawk, and $128\times 128$ImageFruit.
• Figure 5: Condensed images of Fashion-MNIST with 10 images per class.