On Continuity of Robust and Accurate Classifiers

TL;DR

The paper reframes robustness-accuracy tensions in adversarial learning through the lens of continuity, proposing two analytic frameworks—weakly-harmonic and holomorphic hypotheses—and a smoothing-based experimental program to quantify continuity bias. By formulating a gradient-regularized learning objective via Dirichlet energy and extending to complex domains with domains of holomorphy, it derives smoothing feasibility via Cousin problems and analyzes convergence properties of PAC sequences. Empirical results on MNIST-family datasets reveal detectable continuity bias in classification tasks and demonstrate limitations of globally holomorphic classifiers in achieving robust, accurate performance, while highlighting the need for region-wise or modular approaches. The work suggests practical directions (e.g., test-time adaptation, mixture-of-experts) and calls for further theoretical development to understand topological and analytic constraints on robust learning, with significant implications for how continuity is handled in model design and evaluation.

Abstract

The reliability of a learning model is key to the successful deployment of machine learning in various applications. However, it is difficult to describe the phenomenon due to the complicated nature of the problems in machine learning. It has been shown that adversarial training can improve the robustness of the hypothesis. However, this improvement usually comes at the cost of decreased performance on natural samples. Hence, it has been suggested that robustness and accuracy of a hypothesis are at odds with each other. In this paper, we put forth the alternative proposal that it is the continuity of a hypothesis that is incompatible with its robustness and accuracy in many of these scenarios. In other words, a continuous function cannot effectively learn the optimal robust hypothesis. We introduce a framework for a rigorous study of harmonic and holomorphic hypothesis in learning theory terms and provide empirical evidence that continuous hypotheses do not perform as well as discontinuous hypotheses in some common machine learning tasks. From a practical point of view, our results suggests that a robust and accurate learning rule would train different continuous hypotheses for different regions of the domain. From a theoretical perspective, our analysis explains the adversarial examples phenomenon in these situations as a conflict between the continuity of a sequence of functions and its uniform convergence to a discontinuous function. Given that many of the contemporary machine learning models are continuous functions, it is important to theoretically study the continuity of robust and accurate classifiers as it is consequential in their construction, analysis and evaluation.

Figures (9)

• Figure 1: The histogram of the measured continuity bias for MNIST.
• Figure 2: The distribution of the measured continuity bias for FMNIST.
• Figure 3: The distribution of the measured continuity bias for KMNIST.
• Figure 4: The distribution of the measured continuity bias for EMNIST.
• Figure 5: Adversarial examples of MNIST that fell inside the analytic polyhedra of the classifier that was trained on natural samples.

Theorems & Definitions (31)

• Definition 3.1: PAC covering
• Definition 3.2: smoothing problem
• Theorem 3.1
• Definition 3.3: Continuity Bias
• Definition 3.4: PAC Sequence
• Theorem 3.2: Krantz
• Definition 3.5: weakly-harmonic problem
• Theorem 3.3
• Definition 3.6: weakly-harmonic hypothesis space
• Definition 3.7: eigenvalue problem of $\mathcal{H}$