Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unraveling the Black Box of Neural Networks: A Dynamic Extremum Mapper

Shengjian Chen

TL;DR

By reframing neural networks as systems that generalize through dynamic mapping to the extrema of the model function, the paper provides a mathematical lens on generalization. It introduces the extremum-increment (EI) algorithm, which derives parameters by solving homogeneous linear systems and updates layers iteratively, contrasting with backpropagation. The analysis connects extrema dynamics to gradient vanishing/explosion, overfitting, and robustness via weakened termination and surface-neighborhood concepts, offering practical relaxations that can complement gradient-based methods. Overall, the work lays a foundational framework that could inform alternative training paradigms and deepen theoretical understanding of neural network generalization.

Abstract

We point out that neural networks are not black boxes, and their generalization stems from the ability to dynamically map a dataset to the extrema of the model function. We further prove that the number of extrema in a neural network is positively correlated with the number of its parameters. We then propose a new algorithm that is significantly different from back-propagation algorithm, which mainly obtains the values of parameters by solving a system of linear equations. Some difficult situations, such as gradient vanishing and overfitting, can be simply explained and dealt with in this framework.

Unraveling the Black Box of Neural Networks: A Dynamic Extremum Mapper

TL;DR

By reframing neural networks as systems that generalize through dynamic mapping to the extrema of the model function, the paper provides a mathematical lens on generalization. It introduces the extremum-increment (EI) algorithm, which derives parameters by solving homogeneous linear systems and updates layers iteratively, contrasting with backpropagation. The analysis connects extrema dynamics to gradient vanishing/explosion, overfitting, and robustness via weakened termination and surface-neighborhood concepts, offering practical relaxations that can complement gradient-based methods. Overall, the work lays a foundational framework that could inform alternative training paradigms and deepen theoretical understanding of neural network generalization.

Abstract

We point out that neural networks are not black boxes, and their generalization stems from the ability to dynamically map a dataset to the extrema of the model function. We further prove that the number of extrema in a neural network is positively correlated with the number of its parameters. We then propose a new algorithm that is significantly different from back-propagation algorithm, which mainly obtains the values of parameters by solving a system of linear equations. Some difficult situations, such as gradient vanishing and overfitting, can be simply explained and dealt with in this framework.

Paper Structure

This paper contains 27 sections, 22 equations, 8 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. General Characteristics of an Ideal Model
  3. Precise Mapping
  4. Weakened Mapping
  5. N Classification to Binary Classification
  6. General Training Process of an Ideal Model
  7. Situations on Neural Networks
  8. Model Decomposition
  9. Extreme Points of the Model
  10. Continuous Optimization of Parameter Combinations
  11. The EI Algorithm
  12. General Training Method
  13. Reduce the Computational Complexity
  14. Reduce the Computational Scale
  15. Deductions
  16. ...and 12 more sections

Figures (8)

  • Figure 1: The fitting curve with a small number of samples.
  • Figure 2: The fitting curve with a local maximum.
  • Figure 3: The fitting curve with more local maxima/minima.
  • Figure 4: The approximately fitting curve with extended essence.
  • Figure 5: The ideal fitting curve for a binary classification problem.
  • ...and 3 more figures