Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matching

Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, Zhiqiang Shen

TL;DR

Generalized Various Backbone and Statistical Matching (G-VBSM) is the first algorithm to obtain strong performance across both small-scale and large-scale datasets, and surpasses all SOTA methods by margins.

Abstract

The lightweight "local-match-global" matching introduced by SRe2L successfully creates a distilled dataset with comprehensive information on the full 224x224 ImageNet-1k. However, this one-sided approach is limited to a particular backbone, layer, and statistics, which limits the improvement of the generalization of a distilled dataset. We suggest that sufficient and various "local-match-global" matching are more precise and effective than a single one and has the ability to create a distilled dataset with richer information and better generalization. We call this perspective "generalized matching" and propose Generalized Various Backbone and Statistical Matching (G-VBSM) in this work, which aims to create a synthetic dataset with densities, ensuring consistency with the complete dataset across various backbones, layers, and statistics. As experimentally demonstrated, G-VBSM is the first algorithm to obtain strong performance across both small-scale and large-scale datasets. Specifically, G-VBSM achieves a performance of 38.7% on CIFAR-100 with 128-width ConvNet, 47.6% on Tiny-ImageNet with ResNet18, and 31.4% on the full 224x224 ImageNet-1k with ResNet18, under images per class (IPC) 10, 50, and 10, respectively. These results surpass all SOTA methods by margins of 3.9%, 6.5%, and 10.1%, respectively.

Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matching

TL;DR

Generalized Various Backbone and Statistical Matching (G-VBSM) is the first algorithm to obtain strong performance across both small-scale and large-scale datasets, and surpasses all SOTA methods by margins.

Abstract

The lightweight "local-match-global" matching introduced by SRe2L successfully creates a distilled dataset with comprehensive information on the full 224x224 ImageNet-1k. However, this one-sided approach is limited to a particular backbone, layer, and statistics, which limits the improvement of the generalization of a distilled dataset. We suggest that sufficient and various "local-match-global" matching are more precise and effective than a single one and has the ability to create a distilled dataset with richer information and better generalization. We call this perspective "generalized matching" and propose Generalized Various Backbone and Statistical Matching (G-VBSM) in this work, which aims to create a synthetic dataset with densities, ensuring consistency with the complete dataset across various backbones, layers, and statistics. As experimentally demonstrated, G-VBSM is the first algorithm to obtain strong performance across both small-scale and large-scale datasets. Specifically, G-VBSM achieves a performance of 38.7% on CIFAR-100 with 128-width ConvNet, 47.6% on Tiny-ImageNet with ResNet18, and 31.4% on the full 224x224 ImageNet-1k with ResNet18, under images per class (IPC) 10, 50, and 10, respectively. These results surpass all SOTA methods by margins of 3.9%, 6.5%, and 10.1%, respectively.
Paper Structure (12 sections, 11 equations, 5 figures, 1 table)

This paper contains 12 sections, 11 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Method
  4. Data Densification
  5. Technical Detail.
  6. Generalized Backbone Matching
  7. Generalized Statistical Matching
  8. Loss Function in the Evaluation Phase.
  9. Experiment
  10. Hyperparameter Settings.
  11. Ablation Studies
  12. The Trade-off between Efficiency and Effectiveness.

Figures (5)

  • Figure 1: Left: Our proposed G-VBSM consists of three novel and effective modules, named DD, GSM and GBM. The richness and quality of information in the synthetic data have been significantly enhanced compared with the baseline SRe2L through the sequential merging of DD, GBM, and GSM. Right: G-VBSM prioritizes "generalized matching" to ensure consistency between distilled and complete datasets across various backbones, layers, and statistics, and achieves the highest accuracy 31.4% on ImageNet-1k under IPC 10.
  • Figure 2: The overview of G-VBSM on the full 224$\times$224 ImageNet-1k, which ensures the consistency between the distilled and the complete datasets across various backbones, layers and statistics via "generalized matching".
  • Figure 3: Visualization of the mean cosine similarity between pairwise samples within the same class on ImageNet-1k under IPC 10.
  • Figure 4: The illustration of the original loop and the reorder loop.
  • Figure 5: Comparison of the effectiveness and efficiency of G-VBSM components. Among them, "DD+GBM+GSM ($\beta_\textrm{dr}$=0.0)+(MSE+GT)" represents the comprehensive G-VBSM.