Utilizing Class Separation Distance for the Evaluation of Corruption Robustness of Machine Learning Classifiers

TL;DR

This work addresses the problem of evaluating corruption robustness in ML classifiers with a dataset-aware, interpretable metric. It introduces the Minimal Separation Corruption Robustness (MSCR) metric, which uses a dataset-derived corruption distance $\epsilon_{min} = \tfrac{1}{2} \min_{i,j: y_i \neq y_j} dist(x_i, x_j)$ and measures robustness by augmenting test data with uniform noise up to $\epsilon_{min}$, yielding $MSCR = \frac{Acc_{rob-\epsilon_{min}} - Acc_{clean}}{Acc_{clean}}$. The authors validate MSCR on 2D synthetic datasets and CIFAR-10, analyzing multiple models (RF, 1NN, WideResNet) and varying $\epsilon_{train}$ and $\epsilon_{test}$, showing that higher training noise can improve both robustness and accuracy and that the traditional accuracy-robustness tradeoff is not universal. They demonstrate that optimal robustness does not necessarily align with the same level of corruption used during training, and that simple augmentation can provide meaningful gains, with MSCR offering a concrete, dataset-specific interpretive measure for risk assessment. Overall, MSCR provides a practical benchmark for comparing corruption robustness without requiring predefined corruption models, supporting safer deployment of ML systems in real-world settings.

Abstract

Robustness is a fundamental pillar of Machine Learning (ML) classifiers, substantially determining their reliability. Methods for assessing classifier robustness are therefore essential. In this work, we address the challenge of evaluating corruption robustness in a way that allows comparability and interpretability on a given dataset. We propose a test data augmentation method that uses a robustness distance $ε$ derived from the datasets minimal class separation distance. The resulting MSCR (minimal separation corruption robustness) metric allows a dataset-specific comparison of different classifiers with respect to their corruption robustness. The MSCR value is interpretable, as it represents the classifiers avoidable loss of accuracy due to statistical corruptions. On 2D and image data, we show that the metric reflects different levels of classifier robustness. Furthermore, we observe unexpected optima in classifiers robust accuracy through training and testing classifiers with different levels of noise. While researchers have frequently reported on a significant tradeoff on accuracy when training robust models, we strengthen the view that a tradeoff between accuracy and corruption robustness is not inherent. Our results indicate that robustness training through simple data augmentation can already slightly improve accuracy.

Figures (7)

• Figure 1: A robustness requirement (here: $L_2$-norm balls with maximum distance $\epsilon$) assigned to the data points (stars) of a 2D binary dataset (2 input parameters, 2 classes). The shown classifier is not robust, since its dotted decision boundary violates the robustness requirement. To evaluate this, additional points (dots) are augmented within $\epsilon$ of each original point. On those points, the robust accuracy of the classifier is measured – for this classifier, some errors arise.
• Figure 2: The MSCR concept, demonstrated on 2D test data. Data augmentation is carried out like in Figure 1. The distance ($\epsilon_{min}$) is determined by the minimal distance ($2r$) of original points from different classes (black and grey). This way, augmented points of different classes are still separated and classifiers can be both accurate and robust. The decision boundaries of 3 hypothetical classifiers are shown to demonstrate different levels of robustness and their resulting MSCR value.
• Figure 3: Data points in the binary class 2D dataset.
• Figure 4: Effect of hyperparameter $k$ on robust accuracy and its deviation. 2D dataset, $\epsilon_{train},\epsilon_{test}=0.001$.
• Figure 5: Model comparison on 2D Dataset with regards to clean accuracy and robustness (MSCR): RF versus 1NN model with different $\epsilon_{train}$.