Adaptive Dataset Quantization

TL;DR

ADQ tackles data-hungry deep learning by proposing a bin-aware compression framework that replaces uniform per-bin sampling with adaptive sampling guided by three scores. It defines Representativeness Score $Rep$, Diversity Score $Div$, and Importance Score $IS$ as $\hat{I}_n = \hat{Rep}_n + \hat{Div}_n$, where $\hat{Rep}_n$ and $\hat{Div}_n$ are normalized; sampling proportions follow $ r_n = \alpha \hat{I}_n + (1-\alpha)\frac{N(n)}{\sum_n N(n)}$ and per-bin counts $ q_n = \left\lfloor r_n N(n) \right\rfloor$. RS uses a texture-level method, DS uses contrastive learning, and the final step includes MAE-based patch dropping for lossless compression with only a subset of data. Experiments on CIFAR-10 and ImageNet-1K show average gains around 3 percentage points over prior methods, with strong cross-architecture generalization and efficient training.

Abstract

Contemporary deep learning, characterized by the training of cumbersome neural networks on massive datasets, confronts substantial computational hurdles. To alleviate heavy data storage burdens on limited hardware resources, numerous dataset compression methods such as dataset distillation (DD) and coreset selection have emerged to obtain a compact but informative dataset through synthesis or selection for efficient training. However, DD involves an expensive optimization procedure and exhibits limited generalization across unseen architectures, while coreset selection is limited by its low data keep ratio and reliance on heuristics, hindering its practicality and feasibility. To address these limitations, we introduce a newly versatile framework for dataset compression, namely Adaptive Dataset Quantization (ADQ). Specifically, we first identify the sub-optimal performance of naive Dataset Quantization (DQ), which relies on uniform sampling and overlooks the varying importance of each generated bin. Subsequently, we propose a novel adaptive sampling strategy through the evaluation of generated bins' representativeness score, diversity score and importance score, where the former two scores are quantified by the texture level and contrastive learning-based techniques, respectively. Extensive experiments demonstrate that our method not only exhibits superior generalization capability across different architectures, but also attains state-of-the-art results, surpassing DQ by average 3\% on various datasets.

Figures (6)

• Figure 1: The paradigm of three types of dataset condensation methods. The primary difference between these methods lies in the subset generating process. (a) Dataset Distillation synthesizes unreal dataset, (b) coreset selection employs one-time selection, while (c) dataset quantization utilizes multi-time selection as well as sampling.
• Figure 2: The evaluation of normalized representativeness score, diversity score and importance score on CIFAR-10 krizhevsky2009learning. (a) Ideal Condition allows for the best performance of DQ. (b) (c) (d) are representativeness score (RS), diversity score (DS) and importance score (IS) of generated bins on CIFAR-10, respectively.
• Figure 3: The overall framework of the proposed Adaptive Dataset Quantization (ADQ). Following Dataset Quantization (DQ), we first divide the original dataset $\mathbf{D}$ into m non-overlapping bins $[\mathbf S_1,...\mathbf S_n,...\mathbf S_m]$. Next, an importance evaluation is conducted to calculate representativeness score, diversity score and importance score for $\mathbf{S_n}$. We then employ an adaptive sampling based on the importance score and the number of samples in $\mathbf{S_n}$ to obtain a initial compressed set. Eventually, a patch dropping and reconstruction process via MAE DBLP:conf/cvpr/HeCXLDG22 is used to drop uninformative patches, as detailed in the Appendix.
• Figure 4: The illustration of three types of texture level (TL) curves: High TL, Medium TL and Low TL. These curves represent the distances between the expert and the students in our improved trajectories matching.
• Figure 5: The performance of DM DBLP:conf/wacv/ZhaoB23, DQ DBLP:conf/iccv/ZhouWGPLZYF23 and ADQ on (a) high data keep ratio and (c) low data keep ratio on CIFAR-10; and GC DBLP:conf/alt/IyerKBA21, DQ and ADQ on (b) high data keep ratio and (d) low data keep ratio on ImageNet-1K. The dashed lines in grey in (a) and (b) indicate the results when the data keep ratio is 100%.