Adaptive Dataset Quantization: A New Direction for Dataset Pruning

TL;DR

This work introduces Adaptive Dataset Quantization (ADQ), a preprocessing method to compress datasets by reducing intra-sample content while preserving training performance. It combines per-sample linear symmetric quantization with an adaptive bit-width allocation that assigns higher precision to more quantization-sensitive samples, maintaining a fixed overall compression ratio. Through extensive experiments on CIFAR-10/100 and ImageNet-1K, ADQ outperforms traditional dataset pruning and distillation baselines under equivalent compression, and even yields slight gains over full-precision training at moderate compression. The approach significantly boosts data efficiency for edge deployments and opens avenues for integrating dataset and model quantization in a unified efficiency framework.

Abstract

This paper addresses the challenges of storage and communication costs for large-scale datasets in resource-constrained edge devices by proposing a novel dataset quantization approach to reduce intra-sample redundancy. Unlike traditional dataset pruning and distillation methods that focus on inter-sample redundancy, the proposed method compresses each image by reducing redundant or less informative content within samples while preserving essential features. It first applies linear symmetric quantization to obtain an initial quantization range and scale for each sample. Then, an adaptive quantization allocation algorithm is introduced to distribute different quantization ratios for samples with varying precision requirements, maintaining a constant total compression ratio. The main contributions include: (1) being the first to use limited bits to represent datasets for storage reduction; (2) introducing a dataset-level quantization algorithm with adaptive ratio allocation; and (3) validating the method's effectiveness through extensive experiments on CIFAR-10, CIFAR-100, and ImageNet-1K. Results show that the method maintains model training performance while achieving significant dataset compression, outperforming traditional quantization and dataset pruning baselines under the same compression ratios.

Figures (3)

• Figure 1: The left column displays the original images from the train-set. The center column shows the images reconstructed after 8-bit linear symmetric quantization, while the right column presents the results using 4-bit quantization. As observed, 8-bit quantization introduces negligible visual distortion and effectively preserves the perceptual quality of the original images, indicating that it is nearly lossless for this task. In contrast, 4-bit quantization introduces noticeable degradation; however, the primary visual structures and semantic content of the images remain largely intact and recognizable.
• Figure 2: Visualization of CIFAR-10 images with the top 10 smallest distances (a) and top 10 largest distances (b) under 8-bit quantization. Samples with small distances typically contain clear, large objects and simple backgrounds with prominent features. In contrast, samples with large distances often exhibit blurred objects and complex, textured backgrounds, making them more sensitive to feature loss from quantization.
• Figure 3: Overview of the Adaptive Quantization Framework Each sample is initially quantized and dequantized using a uniform bit-width $b$. Based on the distance between the input gradient before and after quantization, bit-widths are then adaptively assigned—higher for samples with larger deviations, and lower for those with smaller deviations.