Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Computational Math with Neural Networks is Hard

Michael Feischl, Fabian Zehetgruber

TL;DR

This work establishes strong hardness results under $SETH$ for three core computational tasks involving neural-network inputs: quadrature of NN realizations, solving Poisson-type PDEs with NN sources, and NN-encoded matrix-vector multiplication. The authors construct a CNF-to-NN reduction to encode SAT into these tasks and show that any higher-order algorithm would imply a faster SAT solver, even for networks with modest depth (three hidden layers) and bounded weights. They extend the results to multiple domains and dimensions, and demonstrate sharpness by presenting fast quadrature for one-layer networks and numerical evidence that quasi-Monte Carlo methods reach near-optimal convergence. The findings reveal fundamental limits on leveraging neural-network surrogates for high-dimensional computation, while also clarifying the boundary where fast exact quadrature is possible (one-layer nets) and where standard Monte Carlo approaches remain the practical route.

Abstract

We show that under some widely believed assumptions, there are no higher-order algorithms for basic tasks in computational mathematics such as: Computing integrals with neural network integrands, computing solutions of a Poisson equation with neural network source term, and computing the matrix-vector product with a neural network encoded matrix. We show that this is already true for very simple feed-forward networks with at least three hidden layers, bounded weights, bounded realization, and sparse connectivity, even if the algorithms are allowed to access the weights of the network. The fundamental idea behind these results is that it is already very hard to check whether a given neural network represents the zero function. The non-locality of the problems above allow us to reduce the approximation setting to deciding whether the input is zero or not. We demonstrate sharpness of our results by providing fast quadrature algorithms for one-layer networks and giving numerical evidence that quasi-Monte Carlo methods achieve the best possible order of convergence for quadrature with neural networks.

Computational Math with Neural Networks is Hard

TL;DR

This work establishes strong hardness results under for three core computational tasks involving neural-network inputs: quadrature of NN realizations, solving Poisson-type PDEs with NN sources, and NN-encoded matrix-vector multiplication. The authors construct a CNF-to-NN reduction to encode SAT into these tasks and show that any higher-order algorithm would imply a faster SAT solver, even for networks with modest depth (three hidden layers) and bounded weights. They extend the results to multiple domains and dimensions, and demonstrate sharpness by presenting fast quadrature for one-layer networks and numerical evidence that quasi-Monte Carlo methods reach near-optimal convergence. The findings reveal fundamental limits on leveraging neural-network surrogates for high-dimensional computation, while also clarifying the boundary where fast exact quadrature is possible (one-layer nets) and where standard Monte Carlo approaches remain the practical route.

Abstract

We show that under some widely believed assumptions, there are no higher-order algorithms for basic tasks in computational mathematics such as: Computing integrals with neural network integrands, computing solutions of a Poisson equation with neural network source term, and computing the matrix-vector product with a neural network encoded matrix. We show that this is already true for very simple feed-forward networks with at least three hidden layers, bounded weights, bounded realization, and sparse connectivity, even if the algorithms are allowed to access the weights of the network. The fundamental idea behind these results is that it is already very hard to check whether a given neural network represents the zero function. The non-locality of the problems above allow us to reduce the approximation setting to deciding whether the input is zero or not. We demonstrate sharpness of our results by providing fast quadrature algorithms for one-layer networks and giving numerical evidence that quasi-Monte Carlo methods achieve the best possible order of convergence for quadrature with neural networks.

Paper Structure

This paper contains 30 sections, 34 theorems, 132 equations, 3 figures, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Open Question
  4. Outline of the paper
  5. Notation
  6. Fundamental assumptions and definitions
  7. Boolean satisfiability problem (SAT)
  8. Algorithms, computational complexity and models of computation
  9. (Probabilistic) decision and approximation algorithms
  10. Exponential time hypothesis (ETH)
  11. Neural networks
  12. Quadrature of neural networks
  13. SAT problems, numerical quadrature and neural networks
  14. Quadrature on the hypercube
  15. Low-precision setting
  16. ...and 15 more sections

Key Result

Lemma 2.2

Under SETH, we have $\widetilde{s}_k>0$ for all $k\geq 3$ and $\lim_{k \to \infty} \widetilde{s}_k=1$.

Figures (3)

  • Figure 1: The connectivity graph of $\Phi_\alpha$ from Lemma \ref{['lem:CNF_to_nn']} for formulas $\alpha$ of the form $(\gamma_1 x_1 \lor \gamma_2 x_2 \lor \gamma_3 x_3) \land (\gamma_4 x_2 \lor \gamma_5 x_4)$ with $\gamma_i\in \{{\rm id},\lnot\}$.
  • Figure 2: Left: Quadrature error for quasi-Monte Carlo quadrature with Sobol points for randomly initialized networks with input dimension five, three to nine hidden layers, and width $100$ as well as for a network with input dimension $784$, three hidden layers and width $784$ trained on the MNIST dataset. The dashed line represents $\mathcal{O}(1/t)$. Right: Visualization of the linear pieces of the realization $\mathcal{R}_\Phi$ of the networks, where same color means same gradient norm of the linear piece (from top-left to bottom-right: depth 3, depth 6, depth 9, mnist).
  • Figure 3: Left: Quadrature error for quasi-Monte Carlo quadrature with Sobol points for the integrand $f$ from \ref{['eq:rquad']}. The dashed line represents $\mathcal{O}(1/t)$. Right: Example visualization of $f$ in 2D.

Theorems & Definitions (64)

  • Lemma 2.2
  • proof
  • Lemma 2.3
  • proof
  • Lemma 3.1
  • proof
  • Lemma 3.2
  • proof
  • Lemma 3.3
  • proof
  • ...and 54 more