Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruning

TL;DR

TDDS addresses the generalization gaps of snapshot-based dataset pruning by introducing a Temporal Dual-Depth Scoring framework that jointly models training dynamics over time and emphasizes well-generalized samples. The outer level maximizes time-variant gradient contributions across a moving window, while the inner level estimates per-sample contributions via projection onto the accumulated gradient, using KL-based loss differences and EMA efficiency. Empirical results on CIFAR-10/100 and ImageNet-1K show state-of-the-art pruning performance across multiple architectures and strategies, supported by thorough ablations on gradient metrics, loss форм, and coreset training. This approach offers a practical, robust method for constructing coresets that generalize across pruning rates and architectures, with strong resilience to corruption and label noise.

Abstract

Dataset pruning aims to construct a coreset capable of achieving performance comparable to the original, full dataset. Most existing dataset pruning methods rely on snapshot-based criteria to identify representative samples, often resulting in poor generalization across various pruning and cross-architecture scenarios. Recent studies have addressed this issue by expanding the scope of training dynamics considered, including factors such as forgetting event and probability change, typically using an averaging approach. However, these works struggle to integrate a broader range of training dynamics without overlooking well-generalized samples, which may not be sufficiently highlighted in an averaging manner. In this study, we propose a novel dataset pruning method termed as Temporal Dual-Depth Scoring (TDDS), to tackle this problem. TDDS utilizes a dual-depth strategy to achieve a balance between incorporating extensive training dynamics and identifying representative samples for dataset pruning. In the first depth, we estimate the series of each sample's individual contributions spanning the training progress, ensuring comprehensive integration of training dynamics. In the second depth, we focus on the variability of the sample-wise contributions identified in the first depth to highlight well-generalized samples. Extensive experiments conducted on CIFAR and ImageNet datasets verify the superiority of TDDS over previous SOTA methods. Specifically on CIFAR-100, our method achieves 54.51% accuracy with only 10% training data, surpassing random selection by 7.83% and other comparison methods by at least 12.69%.

Figures (8)

• Figure 1: (a) Coresets constructed by snapshot-based dataset pruning tend to fluctuate depending on the epochs selected for data selection. The red and blue points represent the data samples of two coresets constructed with 90% pruning rate at Epoch 10 and 100 from the same training progress, respectively. (b) Snapshot-based coresets fail to generalize to pruning settings, especially pruning rates. The accuracy gap between coresets constructed at Epoch 10 and 100 is increasing as the pruning rate grows. Here, we measure importance using the error vector score, a classic snapshot-based criterion defined in paul2021deep. These scores are obtained by training ResNet-18 on CIFAR-10.
• Figure 2: The pipeline of our proposed TDDS. First, we record individual predicted probabilities of samples $f_{t_i}^{x_n} = f_{\bm{\theta}_{t_i}}(\bm{x}_n)$ during training. Then, we approximate the projected gradient, i.e. the component of gradient $\bm{g}_{t_i}^{x_n} = \bm{g}_{t_i}(\bm{x}_n)$ on accumulated gradient $\bm{G}_{t_i}$ direction, with KL-based loss difference between two adjacent epochs. Next, the projected gradients are used to estimate individual contributions of samples. Finally, the important samples are identified with larger projected gradient variances over time.
• Figure 3: Incorporating sample-wise gradient in the current optimization context. After projecting onto the current accumulated gradient, samples with similar gradients exhibit different behaviors.
• Figure 4: Parameter analysis of range $T$ and window size $K$ on CIFAR-100 with ResNet-18. From left to right, the corresponding pruning rates are $0.3$, $0.5$, $0.7$, $0.8$, and $0.9$.
• Figure 5: Parameter analysis of decay coefficient $\beta$ on CIFAR-100 with ResNet-18. From left to right, the corresponding pruning rates are $0.7$, $0.8$, and $0.9$.