Training Fully Connected Neural Networks is $\exists\mathbb{R}$-Complete

TL;DR

It is proved that algebraic numbers of arbitrarily large degree are required as weights to be able to train some instances to optimality, even if all data points are rational.

Abstract

We consider the problem of finding weights and biases for a two-layer fully connected neural network to fit a given set of data points as well as possible, also known as EmpiricalRiskMinimization. Our main result is that the associated decision problem is $\exists\mathbb{R}$-complete, that is, polynomial-time equivalent to determining whether a multivariate polynomial with integer coefficients has any real roots. Furthermore, we prove that algebraic numbers of arbitrarily large degree are required as weights to be able to train some instances to optimality, even if all data points are rational. Our result already applies to fully connected instances with two inputs, two outputs, and one hidden layer of ReLU neurons. Thereby, we strengthen a result by Abrahamsen, Kleist and Miltzow [NeurIPS 2021]. A consequence of this is that a combinatorial search algorithm like the one by Arora, Basu, Mianjy and Mukherjee [ICLR 2018] is impossible for networks with more than one output dimension, unless $\mathsf{NP}=\exists\mathbb{R}$.

Figures (17)

• Figure 1: A fully connected two-layer neural network as studied in this paper. The symbol inside the hidden neurons expresses the $\mathrm{ReLU}$ activation function.
• Figure 2: The value of $f(\cdot, \Theta)$ is fixed (black part), except for the segment through data point $p$. The red, orange and blue segments are just three out of uncountably many possibilities. Its slope can be used to encode a real-valued variable.
• Figure 3: Two intersecting variable gadgets. The slopes of the blue and the red region encode the values. Point $p$ lies in the intersection of both and can encode a linear relationship between them.
• Figure 4: Data points $p$ and $q$ have different labels in the two output dimensions, enforcing that the slopes of the red and the blue pieces are related via a nonlinear dependency.
• Figure 5: Overview of the global arrangement of the gadgets.

Theorems & Definitions (29)

• Definition 1
• Definition 2
• Theorem 3
• Theorem 4
• Proposition 5: Abrahamsen2021_NeuralNetworks
• Definition 6
• Lemma 7
• proof
• Theorem 8: Abrahamsen2022_ArtGallery
• Theorem 9: Abrahamsen2019_Toolbox