Exploring Variance Reduction in Importance Sampling for Efficient DNN Training

TL;DR

This work tackles the challenge of quantifying variance reduction from importance sampling (IS) during DNN training without incurring prohibitive overhead. It develops a real-time variance-estimation framework based on traces of gradient-variance computed from IS minibatches, introduces an effective minibatch size (EMS) for automatic learning-rate adjustment, and a moving-statistics-based algorithm to estimate per-sample weights via an absolute efficiency metric $\mathcal{S}(W)$. A key innovation is the use of logit-gradient approximations to enable cheaper online estimation while preserving the essential variance characteristics; the EMAIS method then ties EMS to learning-rate scaling to preserve training dynamics. Empirical results on benchmark datasets demonstrate that EMAIS achieves lower gradient variance and higher accuracy than competing IS methods, with training times comparable to standard SGD approaches, and show compatibility with Adam. The approach offers practical benefits for variance-controlled training and suggests potential extensions to distributed settings with reduced communication overhead.

Abstract

Importance sampling is widely used to improve the efficiency of deep neural network (DNN) training by reducing the variance of gradient estimators. However, efficiently assessing the variance reduction relative to uniform sampling remains challenging due to computational overhead. This paper proposes a method for estimating variance reduction during DNN training using only minibatches sampled under importance sampling. By leveraging the proposed method, the paper also proposes an effective minibatch size to enable automatic learning rate adjustment. An absolute metric to quantify the efficiency of importance sampling is also introduced as well as an algorithm for real-time estimation of importance scores based on moving gradient statistics. Theoretical analysis and experiments on benchmark datasets demonstrated that the proposed algorithm consistently reduces variance, improves training efficiency, and enhances model accuracy compared with current importance-sampling approaches while maintaining minimal computational overhead.

Figures (10)

• Figure 1: Transitions of $\mathcal{S}(W)$ during training for CINIC-10 and FMNIST.
• Figure 2: Training loss values (top), test error (middle), and EMS transitions (bottom) for FMNIST. Upper two plots compare three methods: SGD-Uni with fixed minibatch size of $N=128$, EMAIS with a minibatch size of $N=128$, and SGD-Uni with dynamic minibatch size $N=N_\mathrm{ems}$, where $N_\mathrm{ems}$ is reused from the values obtained during EMAIS training with $N=128$ (shown in lower plot).
• Figure 3: Relationship between the full gradient and logit gradient norms for models at different training iterations.
• Figure 4: Comparison of trace variance estimates in \ref{['prop:tr_cov_est']}, computed using either full parameter gradients (plotted on the $y$-axis) or logit gradients (on the $x$-axis), as discussed in \ref{['sec:approx_by_logit']}, for models at training iterations 5000 and 25000. The red lines represent fitted regression lines.
• Figure 5: Scatter plots comparing estimates based on full parameter gradients versus logit-based approximations for (a, b) the effective minibatch size $N_{\mathrm{ems}}$ and (c, d) the efficiency metric $\mathcal{S}(W)$, for models at training iterations 5000 and 25000.

Theorems & Definitions (8)

• Proposition 3.1
• proof
• Proposition 3.2: Optimal importance sampling weight alain2016variance
• Proposition 4.1: Trace of gradient variance under uniform and importance sampling
• proof
• Definition 5.1: Effective minibatch size
• Proposition 5.2
• proof