Do stable neural networks exist for classification problems? -- A new view on stability in AI
Z. N. D. Liu, A. C. Hansen
TL;DR
The paper introduces class stability, a measure $\mathcal{S}^p_{\mathcal{M}}(\overline{f})$ based on the distance to the decision boundary $h^p_{\bar{f}}$, to study stability for discontinuous classification functions. It proves two main results: (i) an Interpolation Theorem showing NNs can interpolate a classification on the $\epsilon$-stable set $\mathcal{M}_{\epsilon}$ while preserving near-original stability, and (ii) a Universal Stability Approximation Theorem ensuring the existence of NN estimators that approximate the target function with stability close to that of the target and with a controllably small mislabelled region. The framework also develops a measure-theoretic view of stability, discusses computability caveats (GHA), and provides explicit stability calculations for canonical sets like cubes and Euclidean balls. Together, these results establish that stable NN approximations exist for classification tasks on compact domains, offering a rigorous lens beyond Lipschitz constants to analyze robustness in AI. The work has potential implications for designing robust classifiers and for theoretical analyses of adversarial stability in discontinuous tasks.
Abstract
In deep learning (DL) the instability phenomenon is widespread and well documented, most commonly using the classical measure of stability, the Lipschitz constant. While a small Lipchitz constant is traditionally viewed as guarantying stability, it does not capture the instability phenomenon in DL for classification well. The reason is that a classification function -- which is the target function to be approximated -- is necessarily discontinuous, thus having an 'infinite' Lipchitz constant. As a result, the classical approach will deem every classification function unstable, yet basic classification functions a la 'is there a cat in the image?' will typically be locally very 'flat' -- and thus locally stable -- except at the decision boundary. The lack of an appropriate measure of stability hinders a rigorous theory for stability in DL, and consequently, there are no proper approximation theoretic results that can guarantee the existence of stable networks for classification functions. In this paper we introduce a novel stability measure $\mathscr{S}(f)$, for any classification function $f$, appropriate to study the stability of discontinuous functions and their approximations. We further prove two approximation theorems: First, for any $ε> 0$ and any classification function $f$ on a \emph{compact set}, there is a neural network (NN) $ψ$, such that $ψ- f \neq 0$ only on a set of measure $< ε$, moreover, $\mathscr{S}(ψ) \geq \mathscr{S}(f) - ε$ (as accurate and stable as $f$ up to $ε$). Second, for any classification function $f$ and $ε> 0$, there exists a NN $ψ$ such that $ψ= f$ on the set of points that are at least $ε$ away from the decision boundary.