Semi-Periodic Activation for Time Series Classification

TL;DR

The paper tackles the lack of activation-function research tailored to time series tasks and argues that properties such as nonlinearity, differentiability, and (semi-)periodicity of the derivative influence performance. It introduces LeakySineLU, a semi-periodic activation defined by $\sigma(x)=\sin^{2}(x)+x$ for $x>0$ and $\frac{\sin^{2}(x)+x}{2}$ otherwise, with derivative $\sigma'(x)=\sin(2x)+1$ for $x>0$ and $\frac{\sin(2x)+1}{2}$ for $x\le0$, handling the kink at 0 via sub-derivatives. Through a comprehensive analysis and a large-scale evaluation on 112 UCR time series datasets using MLP and FCN architectures, LeakySineLU demonstrates the best average ranking among compared activations, often outperforming ReLU and its variants. The study highlights the practical impact of tailoring activations to time series data and motivates future integration of semi-periodic activations into advanced architectures like InceptionTime and ResNet for broader time-series tasks such as forecasting and extrinsic regression.

Abstract

This paper investigates the lack of research on activation functions for neural network models in time series tasks. It highlights the need to identify essential properties of these activations to improve their effectiveness in specific domains. To this end, the study comprehensively analyzes properties, such as bounded, monotonic, nonlinearity, and periodicity, for activation in time series neural networks. We propose a new activation that maximizes the coverage of these properties, called LeakySineLU. We empirically evaluate the LeakySineLU against commonly used activations in the literature using 112 benchmark datasets for time series classification, obtaining the best average ranking in all comparative scenarios.

Figures (9)

• Figure 1: Comparison between different activations. Blue lines represent an input time series, red dashed lines represent the values obtained by applying the activations.
• Figure 2: LeakySineLU defined in Equation \ref{['eq:leakysinelu']} represented in blue, and its derivative defined in Equation \ref{['eq:derivative_leakysinelu']} in red.
• Figure 3: Effects of boundaries on time series feature maps. Red region presents the information lost, which means that $f(x) = 0$ for that values.
• Figure 4: Comparison between LeakySineLU, ReLU, PReLU, and Snake on an MLP network among the 112 equal-length datasets.
• Figure 5: Critical Difference diagram with the average rank of each classifier over the 112 equal-length datasets of the UCR archive. The horizontal black bar denotes that the models connected with it have no statistical difference.

Theorems & Definitions (4)

• definition thmcounterdefinition
• definition thmcounterdefinition
• definition thmcounterdefinition
• definition thmcounterdefinition