Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Predictions Based on Pixel Data: Insights from PDEs and Finite Differences

Elena Celledoni, James Jackaman, Davide Murari, Brynjulf Owren

TL;DR

This paper shows that with relatively small networks, they can represent exactly a class of numerical discretizations of PDEs based on the method of lines, and constructively derive these results by exploiting the connections between discrete convolution and finite difference operators.

Abstract

As supported by abundant experimental evidence, neural networks are state-of-the-art for many approximation tasks in high-dimensional spaces. Still, there is a lack of a rigorous theoretical understanding of what they can approximate, at which cost, and at which accuracy. One network architecture of practical use, especially for approximation tasks involving images, is (residual) convolutional networks. However, due to the locality of the linear operators involved in these networks, their analysis is more complicated than that of fully connected neural networks. This paper deals with approximation of time sequences where each observation is a matrix. We show that with relatively small networks, we can represent exactly a class of numerical discretizations of PDEs based on the method of lines. We constructively derive these results by exploiting the connections between discrete convolution and finite difference operators. Our network architecture is inspired by those typically adopted in the approximation of time sequences. We support our theoretical results with numerical experiments simulating the linear advection, heat, and Fisher equations.

Predictions Based on Pixel Data: Insights from PDEs and Finite Differences

TL;DR

This paper shows that with relatively small networks, they can represent exactly a class of numerical discretizations of PDEs based on the method of lines, and constructively derive these results by exploiting the connections between discrete convolution and finite difference operators.

Abstract

As supported by abundant experimental evidence, neural networks are state-of-the-art for many approximation tasks in high-dimensional spaces. Still, there is a lack of a rigorous theoretical understanding of what they can approximate, at which cost, and at which accuracy. One network architecture of practical use, especially for approximation tasks involving images, is (residual) convolutional networks. However, due to the locality of the linear operators involved in these networks, their analysis is more complicated than that of fully connected neural networks. This paper deals with approximation of time sequences where each observation is a matrix. We show that with relatively small networks, we can represent exactly a class of numerical discretizations of PDEs based on the method of lines. We constructively derive these results by exploiting the connections between discrete convolution and finite difference operators. Our network architecture is inspired by those typically adopted in the approximation of time sequences. We support our theoretical results with numerical experiments simulating the linear advection, heat, and Fisher equations.
Paper Structure (22 sections, 3 theorems, 61 equations, 4 figures, 1 algorithm)

This paper contains 22 sections, 3 theorems, 61 equations, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Residual neural networks for time sequences
  3. Characterization of the vector field
  4. Numerical time integrator
  5. Optimization problem to solve
  6. Error bounds for network-based approximations of PDE solutions
  7. Splitting of the approximation errors
  8. Error analysis for PDEs on a two-dimensional spatial domain
  9. Convolutional layers as finite differences
  10. Error analysis for TEXT based on convolution operations
  11. Improving the stability of predictions
  12. Noise injection
  13. Norm preservation
  14. Numerical experiments
  15. Linear advection equation
  16. ...and 7 more sections

Key Result

Theorem 1

Let $U\in\mathbb{R}^{p\times p}$ and for $L,D_{1},D_{2},...,D_{2I}\in\mathbb{R}^{3\times 3}$. Further, let $F_\theta$ be the parametric map defined by Then, $F_{\theta}$ can represent $F$ for suitably chosen parameters

Figures (4)

  • Figure 1: Visual representation of the prediction provided by the neural network. We consider one specific test initial condition, and snapshots coming from the space-time discretization of the reaction-diffusion equation \ref{['eq:fisher']}. The network and PDE parameters can be found respectively in subsection \ref{['sec:fisher']} and appendix \ref{['se:datagen']}. From the left, we have the initial condition, the true space discretization of the solution at time $40\delta t$, the network prediction, and the difference between the last two matrices.
  • Figure 2: Test errors for the linear advection equation.
  • Figure 3: Test errors for the heat equation.
  • Figure 4: Test errors for the Fisher equation.

Theorems & Definitions (7)

  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Corollary 1
  • proof
  • Remark