Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MILP initialization for solving parabolic PDEs with PINNs

Sirui Li, Federica Bragone, Matthieu Barreau, Kateryna Morozovska

TL;DR

This paper uses a convex optimization model to improve the initialization of the weights in PINNs and accelerate convergence, and investigates two optimization models as a first training step, defined as pre-training, one involving only the boundaries and one including physics.

Abstract

Physics-Informed Neural Networks (PINNs) are a powerful deep learning method capable of providing solutions and parameter estimations of physical systems. Given the complexity of their neural network structure, the convergence speed is still limited compared to numerical methods, mainly when used in applications that model realistic systems. The network initialization follows a random distribution of the initial weights, as in the case of traditional neural networks, which could lead to severe model convergence bottlenecks. To overcome this problem, we follow current studies that deal with optimal initial weights in traditional neural networks. In this paper, we use a convex optimization model to improve the initialization of the weights in PINNs and accelerate convergence. We investigate two optimization models as a first training step, defined as pre-training, one involving only the boundaries and one including physics. The optimization is focused on the first layer of the neural network part of the PINN model, while the other weights are randomly initialized. We test the methods using a practical application of the heat diffusion equation to model the temperature distribution of power transformers. The PINN model with boundary pre-training is the fastest converging method at the current stage.

MILP initialization for solving parabolic PDEs with PINNs

TL;DR

This paper uses a convex optimization model to improve the initialization of the weights in PINNs and accelerate convergence, and investigates two optimization models as a first training step, defined as pre-training, one involving only the boundaries and one including physics.

Abstract

Physics-Informed Neural Networks (PINNs) are a powerful deep learning method capable of providing solutions and parameter estimations of physical systems. Given the complexity of their neural network structure, the convergence speed is still limited compared to numerical methods, mainly when used in applications that model realistic systems. The network initialization follows a random distribution of the initial weights, as in the case of traditional neural networks, which could lead to severe model convergence bottlenecks. To overcome this problem, we follow current studies that deal with optimal initial weights in traditional neural networks. In this paper, we use a convex optimization model to improve the initialization of the weights in PINNs and accelerate convergence. We investigate two optimization models as a first training step, defined as pre-training, one involving only the boundaries and one including physics. The optimization is focused on the first layer of the neural network part of the PINN model, while the other weights are randomly initialized. We test the methods using a practical application of the heat diffusion equation to model the temperature distribution of power transformers. The PINN model with boundary pre-training is the fastest converging method at the current stage.

Paper Structure

This paper contains 12 sections, 19 equations, 9 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. PINNs
  4. Heat diffusion equation
  5. Optimization model formulation
  6. Results and Discussion
  7. Pre-training evaluation
  8. Convergence speed
  9. Evaluation of the pre-training algorithms
  10. Evaluation of the training algorithm
  11. Evaluation of the training
  12. Conclusion

Figures (9)

  • Figure 1: Structure of PINN for the heat diffusion equation.
  • Figure 2: Comparison between the hyperbolic tangent and the saturation function. The $\beta_i$ variable is used for the translation to MLP language.
  • Figure 3: Temperature solution given by the initial weights of the vanilla PINN and PINN using boundary and full pre-training with 32 neurons.
  • Figure 4: Temperature solution given by the initial weights of the vanilla PINN and PINN using boundary and full pre-training with 60 neurons.
  • Figure 5: Comsol reference solution.
  • ...and 4 more figures

Theorems & Definitions (1)

  • Remark 1