Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can neural operators always be continuously discretized?

Takashi Furuya, Michael Puthawala, Maarten V. de Hoop, Matti Lassas

TL;DR

The paper investigates the discretization limits of neural operators between Hilbert spaces, revealing a no-go barrier for approximating general infinite-dimensional diffeomorphisms with finite-dimensional diffeomorphisms. It then identifies a constructive path forward by enforcing strong monotonicity, showing that strongly monotone diffeomorphisms admit continuous finite-dimensional approximations and enabling discretization invariance. For bilipschitz neural operators, the authors show a robust representation as alternating compositions of strongly monotone components plus a simple isometry, allowing local inversions and stable finite-rank discretizations via residual networks. A quantitative framework is developed to bound discretization errors and provide universal approximation results for the discretized operators. Collectively, these results establish a rigorous platform for discretizing and inverting neural operators beyond the classic setting, with direct implications for numerical analysis in infinite-dimensional function spaces.

Abstract

We consider the problem of discretization of neural operators between Hilbert spaces in a general framework including skip connections. We focus on bijective neural operators through the lens of diffeomorphisms in infinite dimensions. Framed using category theory, we give a no-go theorem that shows that diffeomorphisms between Hilbert spaces or Hilbert manifolds may not admit any continuous approximations by diffeomorphisms on finite-dimensional spaces, even if the approximations are nonlinear. The natural way out is the introduction of strongly monotone diffeomorphisms and layerwise strongly monotone neural operators which have continuous approximations by strongly monotone diffeomorphisms on finite-dimensional spaces. For these, one can guarantee discretization invariance, while ensuring that finite-dimensional approximations converge not only as sequences of functions, but that their representations converge in a suitable sense as well. Finally, we show that bilipschitz neural operators may always be written in the form of an alternating composition of strongly monotone neural operators, plus a simple isometry. Thus we realize a rigorous platform for discretization of a generalization of a neural operator. We also show that neural operators of this type may be approximated through the composition of finite-rank residual neural operators, where each block is strongly monotone, and may be inverted locally via iteration. We conclude by providing a quantitative approximation result for the discretization of general bilipschitz neural operators.

Can neural operators always be continuously discretized?

TL;DR

The paper investigates the discretization limits of neural operators between Hilbert spaces, revealing a no-go barrier for approximating general infinite-dimensional diffeomorphisms with finite-dimensional diffeomorphisms. It then identifies a constructive path forward by enforcing strong monotonicity, showing that strongly monotone diffeomorphisms admit continuous finite-dimensional approximations and enabling discretization invariance. For bilipschitz neural operators, the authors show a robust representation as alternating compositions of strongly monotone components plus a simple isometry, allowing local inversions and stable finite-rank discretizations via residual networks. A quantitative framework is developed to bound discretization errors and provide universal approximation results for the discretized operators. Collectively, these results establish a rigorous platform for discretizing and inverting neural operators beyond the classic setting, with direct implications for numerical analysis in infinite-dimensional function spaces.

Abstract

We consider the problem of discretization of neural operators between Hilbert spaces in a general framework including skip connections. We focus on bijective neural operators through the lens of diffeomorphisms in infinite dimensions. Framed using category theory, we give a no-go theorem that shows that diffeomorphisms between Hilbert spaces or Hilbert manifolds may not admit any continuous approximations by diffeomorphisms on finite-dimensional spaces, even if the approximations are nonlinear. The natural way out is the introduction of strongly monotone diffeomorphisms and layerwise strongly monotone neural operators which have continuous approximations by strongly monotone diffeomorphisms on finite-dimensional spaces. For these, one can guarantee discretization invariance, while ensuring that finite-dimensional approximations converge not only as sequences of functions, but that their representations converge in a suitable sense as well. Finally, we show that bilipschitz neural operators may always be written in the form of an alternating composition of strongly monotone neural operators, plus a simple isometry. Thus we realize a rigorous platform for discretization of a generalization of a neural operator. We also show that neural operators of this type may be approximated through the composition of finite-rank residual neural operators, where each block is strongly monotone, and may be inverted locally via iteration. We conclude by providing a quantitative approximation result for the discretization of general bilipschitz neural operators.

Paper Structure

This paper contains 37 sections, 27 theorems, 358 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Related work
  3. Our contributions
  4. Definitions and notation
  5. Elements of functional analysis
  6. A general framework for neural operators
  7. Category theoretic framework for discretization
  8. No-go theorem for discretization of diffeomorphisms on Hilbert spaces
  9. Strongly monotone diffeomorphisms and their approximation on finite-dimensional subspaces
  10. Strongly monotone Nemytskii operators and linear bounded operators and their continuous approximation on finite-dimensional subspaces
  11. Strongly monotone generalized neural operators and their continuous approximation on finite-dimensional subspaces
  12. Bilipschitz neural operators are conditionally strongly monotone diffeomorphisms
  13. Quantitative approximation
  14. Conclusion
  15. Proofs and additional examples
  16. ...and 22 more sections

Key Result

Theorem 1

Let ${\mathcal{A}}$ be an approximation scheme that maps diffeomorphisms $F$ on a Hilbert to a sequence of finite-approximations $F_V$ that are themselves diffeomorphisms. If $F_V$ converges to $F$ as $V\to X$, then ${\mathcal{A}}$ is not continuous, that is, there are maps $F^{(j)}$ that converge t

Figures (1)

  • Figure 1: A figure illustrating the proof ideas for Theorem \ref{['thm:no-go']}. It represents the disconnected components of diffeomorphisms that preserve orientation, notated by $\mathrm{diff}^+$, and reverse orientation, notated, $\mathrm{diff}^-$. The horizontal axis abstractly represents the two disconnected components of $\mathrm{diff}$ for a finite-dimensional vector space $V$. The vertical axis represents the dimension of $V$. Observe how the two components of $\mathrm{diff}$ connect as $\dim(V) \to \infty$, and $V$ becomes a Hilbert space $H$.

Theorems & Definitions (64)

  • Definition 1: $\epsilon_V$ approximators and weak approximators
  • Definition 2: Strongly Monotone
  • Definition 3: Bilipschitz
  • Definition 4: Generalized neural operator layer
  • Theorem 1: No-go Theorem, Informal
  • Definition 5: Category of Hilbert Space Diffeomorphisms
  • Definition 6: Category of Approximation Sequences
  • Definition 7: Approximation Functor
  • Definition 8
  • Theorem 2
  • ...and 54 more