Certified and accurate computation of function space norms of deep neural networks

TL;DR

This work presents a framework for the certified and accurate computation of integral quantities of neural networks, including Lebesgue and Sobolev norms, by combining interval arithmetic enclosures on axis-aligned boxes with adaptive marking/refinement and quadrature-based aggregation.

Abstract

Neural network methods for PDEs require reliable error control in function space norms. However, trained neural networks can typically only be probed at a finite number of point values. Without strong assumptions, point evaluations alone do not provide enough information to derive tight deterministic and guaranteed bounds on function space norms. In this work, we move beyond a purely black-box setting and exploit the neural network structure directly. We present a framework for the certified and accurate computation of integral quantities of neural networks, including Lebesgue and Sobolev norms, by combining interval arithmetic enclosures on axis-aligned boxes with adaptive marking/refinement and quadrature-based aggregation. On each box, we compute guaranteed lower and upper bounds for function values and derivatives, and propagate these local certificates to global lower and upper bounds for the target integrals. Our analysis provides a general convergence theorem for such certified adaptive quadrature procedures and instantiates it for function values, Jacobians, and Hessians, yielding certified computation of $L^p$, $W^{1,p}$, and $W^{2,p}$ norms. We further show how these ingredients lead to practical certified bounds for PINN interior residuals. Numerical experiments illustrate the accuracy and practical behavior of the proposed methods.

Figures (9)

• Figure 1: Overview of the adaptive refinement strategies. Left: Certified relative gap of the proposed algorithms for the computation of $L^p$, $W^{1,p}$, and $W^{2,p}$ norms of a smoothed disc function. The plotted quantity is our upper bound minus the lower bound, normalized by the final norm estimate. Middle: Heatmap of the local $W^{2,p}$ norm error for a deep architecture approximating a smoothed disc function. The partition is computed by $\mathrm{AdaQuad}$. Right: Adaptive partition after 30 refinement steps of $\mathrm{AdaQuad}$ applied to a random ReLU neural network (see Remark \ref{['rem:ExactIntegration']}).
• Figure 2: Left: Boundaries of affine linear pieces for a random ReLU network of width 40 and depth 5. Middle: Adaptive partition after 30 refinement steps of $\mathrm{AdaQuad}$ with color indicating local error bounds. The boundaries of affine linear regions are overlaid. Right: Adaptive partition after 40 refinement steps of $\mathrm{AdaQuad}$ with color indicating local error bounds. The boundaries of affine linear regions are overlaid.
• Figure 3: Mean and $95\%$ confidence interval of the normalized global bound gap for Sobolev norms for 100 untrained tanh networks, refined adaptively, comparing a deep (three hidden layers with 32 neurons) and a wide (one hidden layer with 200 neurons) architecture.
• Figure 4: Mean and $95\%$ confidence interval of the normalized global bound gap for Sobolev norms of 100 tanh networks, refined with Dörfler marking, comparing a deep (three hidden layers with 32 neurons) and a wide (one hidden layer with 200 neurons) architecture. The neural networks are trained to fit the Gaussian peak \ref{['eq:gaussian_peak']} function.
• Figure 5: Mean and $95\%$ confidence interval of the normalized global bound gap for the $L^p$ norm for 100 ReLU networks, comparing a deep and wide architecture. The trained neural networks approximate the one-dimensional Gaussian-peak function \ref{['eq:gaussian_peak']}.

Theorems & Definitions (95)

• Definition 2.1
• Remark 2.2
• Definition 2.3: Feedforward neural network
• Definition 2.4
• Remark 2.5
• Definition 2.6
• Definition 2.7
• Definition 2.8: rump2010verification
• Definition 2.9: rump2010verification
• Proposition 2.10: moore1966interval