Efficient Dataset Distillation through Low-Rank Space Sampling

TL;DR

This work addresses the redundancy and inefficiency in dataset distillation by introducing Matching Training Trajectories with Low-rank Space Sampling (MTT-LSS), which models synthetic data in multiple low-rank subspaces encoded by shared dimension mappers and basis vectors. By representing data as low-rank components and optimizing soft, trainable labels, the approach expands synthetic data diversity without increasing storage costs, improving meta-test accuracy across SVHN, CIFAR-10, and CIFAR-100, especially at low compression rates. The key contributions are the low-rank space sampling framework, the use of shared dimension mappers and basis vectors to reduce redundancy, and the integration of soft labels with a progressive trajectory-matching optimization strategy, culminating in robust improvements over state-of-the-art distillation methods. This method offers a scalable, cost-effective pathway for generating large, diverse synthetic datasets under tight storage constraints, with potential applicability to continual learning, NAS, and privacy-preserving tasks.

Abstract

Huge amount of data is the key of the success of deep learning, however, redundant information impairs the generalization ability of the model and increases the burden of calculation. Dataset Distillation (DD) compresses the original dataset into a smaller but representative subset for high-quality data and efficient training strategies. Existing works for DD generate synthetic images by treating each image as an independent entity, thereby overlooking the common features among data. This paper proposes a dataset distillation method based on Matching Training Trajectories with Low-rank Space Sampling(MTT-LSS), which uses low-rank approximations to capture multiple low-dimensional manifold subspaces of the original data. The synthetic data is represented by basis vectors and shared dimension mappers from these subspaces, reducing the cost of generating individual data points while effectively minimizing information redundancy. The proposed method is tested on CIFAR-10, CIFAR-100, and SVHN datasets, and outperforms the baseline methods by an average of 9.9%.

Figures (5)

• Figure 1: Illustration of the dataset distillation framework based on low-rank space sampling. The synthetic dataset is composed of basis vectors and dimension mappers. The dimension mappers are shared components for multiple basis vectors, preserving some common features of the synthetic dataset and effectively reducing the information redundancy problem.
• Figure 2: Comparison of Conventional Data Distillation Paradigm and Data Augmentation Distillation Paradigm. By employing data augmentation methods, a more compact feature representation can be achieved, enabling the generation of a larger number of synthetic images at the same storage cost.
• Figure 3: Using a step-by-step optimization strategy enables synthetic data to learn simple features at the beginning of training, and gradually increases the difficulty of matching trajectories as the feature representation capability of synthetic data increases.
• Figure 4: Dimension Mapping Component, Basis Vectors, and Visualized Results of the Synthesized Images (the lower part shows the results obtained by the baseline method). The low-rank sampling method based on the shared matrix can obtain more synthetic data at the same storage cost.
• Figure 5: Comparison of synthesized images with different low-rank values $r$. Higher $r$ improves details but increases storage cost.