Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep Learning Optimization Using Self-Adaptive Weighted Auxiliary Variables

Yaru Liu, Yiqi Gu, Michael K. Ng

TL;DR

This work tackles the difficulty of training deep networks for least-squares learning due to non-convexity and vanishing gradients. It introduces self-adaptive weighted auxiliary variables (SAPM) to reformulate the optimization so that minimizing the auxiliary-variable loss remains consistent with the original mean squared loss, with proven bounds for FNNs and PINNs. The authors develop SAPM-FNN and SAPM-PINN alongside alternating-direction optimization, and demonstrate through extensive experiments that SAPM achieves higher accuracy and robustness than traditional LS and penalty-based approaches, especially for deeper networks. The approach offers a principled path to reliable deep learning in regression settings and PDE-informed modeling, with potential extensions to broader architectures and higher-order problems.

Abstract

In this paper, we develop a new optimization framework for the least squares learning problem via fully connected neural networks or physics-informed neural networks. The gradient descent sometimes behaves inefficiently in deep learning because of the high non-convexity of loss functions and the vanishing gradient issue. Our idea is to introduce auxiliary variables to separate the layers of the deep neural networks and reformulate the loss functions for ease of optimization. We design the self-adaptive weights to preserve the consistency between the reformulated loss and the original mean squared loss, which guarantees that optimizing the new loss helps optimize the original problem. Numerical experiments are presented to verify the consistency and show the effectiveness and robustness of our models over gradient descent.

Deep Learning Optimization Using Self-Adaptive Weighted Auxiliary Variables

TL;DR

This work tackles the difficulty of training deep networks for least-squares learning due to non-convexity and vanishing gradients. It introduces self-adaptive weighted auxiliary variables (SAPM) to reformulate the optimization so that minimizing the auxiliary-variable loss remains consistent with the original mean squared loss, with proven bounds for FNNs and PINNs. The authors develop SAPM-FNN and SAPM-PINN alongside alternating-direction optimization, and demonstrate through extensive experiments that SAPM achieves higher accuracy and robustness than traditional LS and penalty-based approaches, especially for deeper networks. The approach offers a principled path to reliable deep learning in regression settings and PDE-informed modeling, with potential extensions to broader architectures and higher-order problems.

Abstract

In this paper, we develop a new optimization framework for the least squares learning problem via fully connected neural networks or physics-informed neural networks. The gradient descent sometimes behaves inefficiently in deep learning because of the high non-convexity of loss functions and the vanishing gradient issue. Our idea is to introduce auxiliary variables to separate the layers of the deep neural networks and reformulate the loss functions for ease of optimization. We design the self-adaptive weights to preserve the consistency between the reformulated loss and the original mean squared loss, which guarantees that optimizing the new loss helps optimize the original problem. Numerical experiments are presented to verify the consistency and show the effectiveness and robustness of our models over gradient descent.
Paper Structure (22 sections, 4 theorems, 79 equations, 3 figures, 8 tables, 2 algorithms)

This paper contains 22 sections, 4 theorems, 79 equations, 3 figures, 8 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contributions
  3. Organization
  4. Fully-connected Neural Networks
  5. Least Squares Problems
  6. Models with Auxiliary Variables
  7. Self-adaptive Weighted Auxiliary Variables
  8. Implementation
  9. Physics-informed Neural Networks
  10. Least Squares Models for First-Order Linear Equations
  11. Models with Auxiliary Variables
  12. Self-adaptive Weighted Auxiliary Variables
  13. Implementation
  14. Numerical Experiments
  15. Learning Problems with FNNs
  16. ...and 7 more sections

Key Result

Theorem 1

Suppose $\sigma$ is Lipschitz continuous, i.e., $|\sigma(z_1)-\sigma(z_2)|\leq B|z_1-z_2|$ for some $B>0$ and any $z_1,z_2\in\mathbb{R}$. Given $\mathcal{L}$ and $\mathcal{L}_{\rm{S}}$ defined in LS_FNN and SAPM_FNN, respectively. Then for all $\{W_l,b_l\}_{l=1}^L$ and $\{a_l\}_{l=1}^{L-1}$, it hold where $C_{B,\beta}=\max\{1,B^{2L-2}\}\cdot\max_{l=1,\dots,L-1}\{1,\beta_l^{-1}\}$.

Figures (3)

  • Figure 1: The loss versus iterations in learning $f(x)=\sin(x^2)$. (Results are from 10 random seeds.)
  • Figure 2: The actual loss $\mathcal{L}_{\rm{P}}$/$\mathcal{L}_{\rm{S}}$ (solid curve) and the corresponding mean squared loss $\mathcal{L}$ (dashed curve) versus iterations in learning $f(x)=\sin(x^2)$. (Results are from the best seed.)
  • Figure 3: Partial plots of the learner $\phi(x;\theta)$, target function $f(x)=\sin(x^2)$ and training points $\{(x_n,y_n)\}$ in a test of SAPM.

Theorems & Definitions (8)

  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Theorem 3
  • proof
  • Theorem 4
  • proof