Padé Neurons for Efficient Neural Models

TL;DR

The paper introduces Padé approximant neurons (Paons), an inherently non-linear neural unit that computes outputs as a ratio of polynomials and can subsume prior neuron models. It couples Paons with two Shifter variants (kernel-wise and element-wise) to expand receptive fields, forming Padé Layers (PaLa) suitable for both fully-connected and convolutional networks. Paon^S, a smoothed variant, stabilizes training by preventing denominator singularities, while preserving stronger non-linearity and avoiding fixed activations. Across image super-resolution, compression, and classification, replacing standard convolutions with PaLa yields improved or comparable performance with fewer layers, and Paons show resilience to lower-precision implementations, suggesting practical deployment advantages. The approach unifies and extends previous non-linear neuron models, highlights efficiency gains, and outlines trade-offs related to implementation complexity and training speed.

Abstract

Neural networks commonly employ the McCulloch-Pitts neuron model, which is a linear model followed by a point-wise non-linear activation. Various researchers have already advanced inherently non-linear neuron models, such as quadratic neurons, generalized operational neurons, generative neurons, and super neurons, which offer stronger non-linearity compared to point-wise activation functions. In this paper, we introduce a novel and better non-linear neuron model called Padé neurons (Paons), inspired by Padé approximants. Paons offer several advantages, such as diversity of non-linearity, since each Paon learns a different non-linear function of its inputs, and layer efficiency, since Paons provide stronger non-linearity in much fewer layers compared to piecewise linear approximation. Furthermore, Paons include all previously proposed neuron models as special cases, thus any neuron model in any network can be replaced by Paons. We note that there has been a proposal to employ the Padé approximation as a generalized point-wise activation function, which is fundamentally different from our model. To validate the efficacy of Paons, in our experiments, we replace classic neurons in some well-known neural image super-resolution, compression, and classification models based on the ResNet architecture with Paons. Our comprehensive experimental results and analyses demonstrate that neural models built by Paons provide better or equal performance than their classic counterparts with a smaller number of layers. The PyTorch implementation code for Paon is open-sourced at https://github.com/onur-keles/Paon.

Figures (8)

• Figure 1: Illustration of a Padé neuron ($\operatorname{\textit{Paon}}$) for $[K/L] = [2/3]$, where $a_{0}$ is bias for numerator, $(\cdot)^{k}$ takes $k^{\text{th}}$ power of the input in element-wise manner, $\frac{(\cdot)}{(\cdot)}$ implements Eq. \ref{['eq:pade_neuron_smooth']}. The $\operatorname{Shifter}$ module shifts the input features when $\circledast$ is convolution.
• Figure 2: The model architecture for the super-resolution experiments. A shallow feature extractor layer is followed by a series of $\operatorname{\textit{PaLa}}$ blocks. The refined features are added to the initial extracted features to form an image in the desired resolution.
• Figure 3: The structure of residual blocks (RB) and wide residual blocks (WRB). For WRB, $w>1$, while for RB, $w=1$.
• Figure 4: Number of times the denominator $Q_{L}(n_1,n_2)$ is close to $0$ vs. number of training iterations. The first, second, and third rows show the plots for the first, second and third residual blocks, respectively. The first and second columns correspond to the first and second layers in each residual block.
• Figure 5: Visual comparisons on img_024, img_073 and img_076 from Urban100 dataset for $\times4$ SR. Crop-outs from left to right, top row: ground truth, PadéNet, SelfONN, SuperONN, bottom row: ResNet, $\operatorname{PAU}$-Net, DCN $1\times1$ DCN, $3\times3$.