From Dead Neurons to Deep Approximators: Deep Bernstein Networks as a Provable Alternative to Residual Layers
Ibrahim Albool, Malak Gamal El-Din, Salma Elmalaki, Yasser Shoukry
TL;DR
The paper tackles vanishing gradients and dead neurons in deep networks by replacing standard activations with learnable Bernstein polynomials, formulating DeepBern-Nets as residual-free architectures. It provides a theoretical foundation showing a lower bound on Bernstein derivatives, ensuring gradient persistence, and an architectural re-parameterization plus batch normalization to stabilize training. It further proves that approximation error decays exponentially with depth, $\|\mathcal{N}-f\|_\infty \le C_d \cdot \omega_f(1/n^L)$, outperforming ReLU-based polynomial rates, and corroborates these claims with experiments on HIGGS and MNIST demonstrating strong performance without skip connections. Collectively, the work offers a principled path toward deep, residual-free networks with enhanced expressive capacity and training stability, potentially enabling more efficient and scalable models. The results imply that Bernstein activations can deliver both high representational density and trainability in deep architectures. $$
Abstract
Residual connections are the de facto standard for mitigating vanishing gradients, yet they impose structural constraints and fail to address the inherent inefficiencies of piecewise linear activations. We show that Deep Bernstein Networks (which utilizes Bernstein polynomials as activation functions) can act as residual-free architecture while simultaneously optimize trainability and representation power. We provide a two-fold theoretical foundation for our approach. First, we derive a theoretical lower bound on the local derivative, proving it remains strictly bounded away from zero. This directly addresses the root cause of gradient stagnation; empirically, our architecture reduces ``dead'' neurons from 90\% in standard deep networks to less than 5\%, outperforming ReLU, Leaky ReLU, SeLU, and GeLU. Second, we establish that the approximation error for Bernstein-based networks decays exponentially with depth, a significant improvement over the polynomial rates of ReLU-based architectures. By unifying these results, we demonstrate that Bernstein activations provide a superior mechanism for function approximation and signal flow. Our experiments on HIGGS and MNIST confirm that Deep Bernstein Networks achieve high-performance training without skip-connections, offering a principled path toward deep, residual-free architectures with enhanced expressive capacity.