Robust Deep Network Learning of Nonlinear Regression Tasks by Parametric Leaky Exponential Linear Units (LELUs) and a Diffusion Metric

TL;DR

The paper tackles robust learning of multidimensional nonlinear regression with deep networks by introducing a parametric activation function, Leaky ELU (LELU), equipped with a trainable negative-slope parameter $\beta$ to control activation flexibility. It pairs this with a diffusion-loss metric based on a modified Laplacian that quantifies entropy-in-features and overfitting without data splits, enabling internal monitoring of generalization. Across canonical 1D nonlinear tasks and a complex 3D electric-motor model, LELU demonstrates improved generalization and reduced sensitivity to model size compared to traditional activations like ReLU, ELU, SiLU, and Softplus, often via favorable diffusion scores rather than solely minimizing training MAE. The work suggests that LELU acts as an implicit regularizer through its smoothness and controllable flexibility, while the diffusion metric provides a practical tool for detecting overfitting and guiding model configuration in regression problems.

Abstract

This document proposes a parametric activation function (ac.f.) aimed at improving multidimensional nonlinear data regression. It is a established knowledge that nonlinear ac.f's are required for learning nonlinear datasets. This work shows that smoothness and gradient properties of the ac.f. further impact the performance of large neural networks in terms of overfitting and sensitivity to model parameters. Smooth but vanishing-gradient ac.f's such as ELU or SiLU (Swish) have limited performance and non-smooth ac.f's such as RELU and Leaky-RELU further impart discontinuity in the trained model. Improved performance is demonstrated with a smooth "Leaky Exponential Linear Unit", with non-zero gradient that can be trained. A novel diffusion-loss metric is also proposed to gauge the performance of the trained models in terms of overfitting.

Figures (14)

• Figure 1: Graph of several activation functions.
• Figure 2: A vanilla feedforward neural network of depth $n$ and width $m$, with three inputs and one output. The output layer is linear.
• Figure 3: A one-dimensional mesh: the function values are shown by squares and the mesh nodes by circles. The original training mesh is shown as black nodes and the created staggered nodes, in red with half indices. Brown and orange squares show possible oscillations around node $i$.
• Figure 4: The original multidimensional training mesh shown as black points and the created cell-centred staggered points shown in red, with half indices. The diffusion sensors are computed along the centre-crossing diagonals shown in red. Filled symbols are used for the sensor computations for the central node ($(i,j,k)$ in 3d, for instance).
• Figure 5: Trained hyperbolic tangent function plots for the different activation functions for the He-Normal initialisation (no seed).