Brownian ReLU(Br-ReLU): A New Activation Function for a Long-Short Term Memory (LSTM) Network

TL;DR

The paper addresses gradient instability and dying-neuron issues when using conventional activations on noisy financial time series by introducing Brownian ReLU (Br-ReLU), a stochastic activation inspired by Brownian motion for LSTMs. Br-ReLU preserves the positive-branch identity while modeling negative inputs through Monte Carlo estimates with a learnable scale parameter \alpha, facilitating adaptive stochasticity and smoother gradient propagation. The authors define the activation, derive gradients with respect to \alpha, outline a Monte Carlo–based training procedure, and integrate Br-ReLU into an LSTM cell. Empirical results on Apple, Ghana Commercial Bank, and S&P 500 data (and LendingClub classification) show Br-ReLU achieving lower MSE and higher R^2 than baseline activations, with alpha-adaptation enabling data-driven control of the stochastic perturbation; in classification tasks, Br-ReLU offers competitive performance, though ROC-AUC improvements are modest and hinge on selecting appropriate \alpha. Overall, Br-ReLU provides a robust, adaptive activation for financial forecasting that enhances gradient flow and generalization in stochastic environments, with practical implications for risk analytics and time-series modeling.

Abstract

Deep learning models are effective for sequential data modeling, yet commonly used activation functions such as ReLU, LeakyReLU, and PReLU often exhibit gradient instability when applied to noisy, non-stationary financial time series. This study introduces BrownianReLU, a stochastic activation function induced by Brownian motion that enhances gradient propagation and learning stability in Long Short-Term Memory (LSTM) networks. Using Monte Carlo simulation, BrownianReLU provides a smooth, adaptive response for negative inputs, mitigating the dying ReLU problem. The proposed activation is evaluated on financial time series from Apple, GCB, and the S&P 500, as well as LendingClub loan data for classification. Results show consistently lower Mean Squared Error and higher $R^2$ values, indicating improved predictive accuracy and generalization. Although ROC-AUC metric is limited in classification tasks, activation choice significantly affects the trade-off between accuracy and sensitivity, with Brownian ReLU and the selected activation functions yielding practically meaningful performance.

Figures (6)

• Figure 1: Illustrative Monte Carlo paths for Br-ReLU Activation Function
• Figure 2: Varying values of $\alpha$ at M=200
• Figure 3: Varying values of $\alpha$ at M=500
• Figure 4: Varying values of $\alpha$ at M=1000
• Figure 5: Varying values of $\alpha$ at M=1500