Computability of Classification and Deep Learning: From Theoretical Limits to Practical Feasibility through Quantization
Holger Boche, Vit Fojtik, Adalbert Fono, Gitta Kutyniok
TL;DR
This work analyzes fundamental computability limits in deep learning, showing that in the real-valued setting both classification (Type 1) and training (Type 2) face intrinsic noncomputability barriers. It establishes that classifier computability aligns with the semi-decidability of class regions, and proves that no universal data-driven learner can reconstruct every computable network from data. The authors then present strategies to circumvent these limits, notably by allowing approximation and by moving to quantized (fixed-point) models, which can restore computability for both learning and classification in many practical cases, though the quantization function itself can be noncomputable. The results highlight that ground-truth problem structure and discrete representations are key to achieving trustworthy, computable guarantees in DL, while underscoring limits that motivate exploring non-digital or specialized hardware approaches for safety-critical applications.
Abstract
The unwavering success of deep learning in the past decade led to the increasing prevalence of deep learning methods in various application fields. However, the downsides of deep learning, most prominently its lack of trustworthiness, may not be compatible with safety-critical or high-responsibility applications requiring stricter performance guarantees. Recently, several instances of deep learning applications have been shown to be subject to theoretical limitations of computability, undermining the feasibility of performance guarantees when employed on real-world computers. We extend the findings by studying computability in the deep learning framework from two perspectives: From an application viewpoint in the context of classification problems and a general limitation viewpoint in the context of training neural networks. In particular, we show restrictions on the algorithmic solvability of classification problems that also render the algorithmic detection of failure in computations in a general setting infeasible. Subsequently, we prove algorithmic limitations in training deep neural networks even in cases where the underlying problem is well-behaved. Finally, we end with a positive observation, showing that in quantized versions of classification and deep network training, computability restrictions do not arise or can be overcome to a certain degree.