Measuring training variability from stochastic optimization using robust nonparametric testing

TL;DR

This work addresses run-to-run variability in deep neural network training arising from stochastic optimization by introducing a robust, nonparametric testing framework based on $α$-trimming. It defines a reference distribution from ensembles of trained models and develops a trimming-based two-sample KS test to compare a candidate model to this reference, yielding a practical discrepancy measure $\hat{α}$. The approach provides a richer, distribution-focused assessment than traditional metrics like accuracy, churn, or calibration error, and is demonstrated on CNN and ViT transfer-learning tasks to guide ensemble sizing and seed selection. The method offers principled, scalable guidance for selecting seeds and constructing reliable ensembles, with potential extensions to multi-class problems and alternative distributional distances.

Abstract

Deep neural network training often involves stochastic optimization, meaning each run will produce a different model. This implies that hyperparameters of the training process, such as the random seed itself, can potentially have significant influence on the variability in the trained models. Measuring model quality by summary statistics, such as test accuracy, can obscure this dependence. We propose a robust hypothesis testing framework and a novel summary statistic, the $α$-trimming level, to measure model similarity. Applying hypothesis testing directly with the $α$-trimming level is challenging because we cannot accurately describe the distribution under the null hypothesis. Our framework addresses this issue by determining how closely an approximate distribution resembles the expected distribution of a group of individually trained models and using this approximation as our reference. We then use the $α$-trimming level to suggest how many training runs should be sampled to ensure that an ensemble is a reliable representative of the true model performance. We also show how to use the $α$-trimming level to measure model variability and demonstrate experimentally that it is more expressive than performance metrics like validation accuracy, churn, or expected calibration error when taken alone. An application of fine-tuning over random seed in transfer learning illustrates the advantage of our new metric.

Figures (17)

• Figure 1: Hypothetical decision boundaries corresponding to two models with the same accuracy. The shaded regions represent the underlying data distribution.
• Figure 2: The experiment design. Randomness is used within the training algorithm for initialization and batch selection. We train (or fine-tune) $M$ models independently using the same training data. Each model is then evaluated on the test set. The resulting values are used to form empirical CDFs.
• Figure 3: Illustration of the parameters in the trimming-based two-sample test. The null hypothesis uses the $L_{1}$-contamination ball, $\mathcal{B}_1( \bar{F}_{\pi|\mathcal{D}_{\mathrm{param}}},\nu)$. The test will accept if the $L_{\infty}$ distance between $\bar{F}_{\pi|\mathcal{D}_{\mathrm{param}}}$ and the set of $\alpha$-trimmings of $\hat{G}_{0}$, $\mathcal{R}_{\alpha}(\hat{G}_{0})$, is small.
• Figure 4: (Left) Histogram of logit gaps from the ensemble model with the upper and lower envelopes representing the maximum and minimum probability attained in each bin among individual candidate models. (Right) A plot showing the evolution of validation accuracy of CNN models over different epochs. The solid red dots represent the mean validation accuracy over 800 seeds at each epoch, the light-colored area denotes one standard deviation, and the purple area represents the maximum and minimum values at that epoch.
• Figure 5: A plot showing $L_{\infty}$-distance of the eCDF of ensemble models $\hat{\bar{H}}'$ (formed with $M_{\mathrm{ens}}$ models) w.r.t. the reference $\hat{\bar{G}}$ (formed with $M$ models) against validation accuracy of ensemble models. For each value of $M_{\mathrm{ens}}$, we choose $M_{\mathrm{ens}}$ candidate models to form one ensemble model and repeat this experiment 500 times through sampling with replacement to create 500 ensemble models. Thus, each dot with a fixed color represents one out of these 500 ensemble models and each color indicates the value of $M_{\mathrm{ens}}$ or the number of candidate models in the pool. We chose $\epsilon=0.01$, to compute the threshold $\delta_{a}$ in \ref{['eq: Two samp threshold']}.

Theorems & Definitions (5)

• Remark 1
• Theorem 1
• Corollary 1
• Lemma 1: barrio2020box, Lemma 2.4
• Theorem 2: barrio2020box, Theorem 2.5