Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Sequential Model Performance with Squared Sigmoid TanH (SST) Activation Under Data Constraints

Barathi Subramanian, Rathinaraja Jeyaraj, Anand Paul

TL;DR

The paper addresses the difficulty of modeling sparse temporal patterns with vanishing gradients in sequential models trained on small datasets. It proposes SST, which squares the outputs of Sigmoid and TanH activations to amplify strong signals and preserve gradient flow, improving memory and representation across time. Across diverse tasks—sign language recognition, gait classification, human activity recognition, and gold-price forecasting—SST-enhanced GRUs/LSTMs achieve higher test accuracy, improved AUC, and lower MSE compared with baselines, demonstrating broad applicability under data constraints. The approach offers a general, activation-level modification that improves learning efficiency and predictive performance in real-world temporal applications.

Abstract

Activation functions enable neural networks to learn complex representations by introducing non-linearities. While feedforward models commonly use rectified linear units, sequential models like recurrent neural networks, long short-term memory (LSTMs) and gated recurrent units (GRUs) still rely on Sigmoid and TanH activation functions. However, these classical activation functions often struggle to model sparse patterns when trained on small sequential datasets to effectively capture temporal dependencies. To address this limitation, we propose squared Sigmoid TanH (SST) activation specifically tailored to enhance the learning capability of sequential models under data constraints. SST applies mathematical squaring to amplify differences between strong and weak activations as signals propagate over time, facilitating improved gradient flow and information filtering. We evaluate SST-powered LSTMs and GRUs for diverse applications, such as sign language recognition, regression, and time-series classification tasks, where the dataset is limited. Our experiments demonstrate that SST models consistently outperform RNN-based models with baseline activations, exhibiting improved test accuracy.

Enhancing Sequential Model Performance with Squared Sigmoid TanH (SST) Activation Under Data Constraints

TL;DR

The paper addresses the difficulty of modeling sparse temporal patterns with vanishing gradients in sequential models trained on small datasets. It proposes SST, which squares the outputs of Sigmoid and TanH activations to amplify strong signals and preserve gradient flow, improving memory and representation across time. Across diverse tasks—sign language recognition, gait classification, human activity recognition, and gold-price forecasting—SST-enhanced GRUs/LSTMs achieve higher test accuracy, improved AUC, and lower MSE compared with baselines, demonstrating broad applicability under data constraints. The approach offers a general, activation-level modification that improves learning efficiency and predictive performance in real-world temporal applications.

Abstract

Activation functions enable neural networks to learn complex representations by introducing non-linearities. While feedforward models commonly use rectified linear units, sequential models like recurrent neural networks, long short-term memory (LSTMs) and gated recurrent units (GRUs) still rely on Sigmoid and TanH activation functions. However, these classical activation functions often struggle to model sparse patterns when trained on small sequential datasets to effectively capture temporal dependencies. To address this limitation, we propose squared Sigmoid TanH (SST) activation specifically tailored to enhance the learning capability of sequential models under data constraints. SST applies mathematical squaring to amplify differences between strong and weak activations as signals propagate over time, facilitating improved gradient flow and information filtering. We evaluate SST-powered LSTMs and GRUs for diverse applications, such as sign language recognition, regression, and time-series classification tasks, where the dataset is limited. Our experiments demonstrate that SST models consistently outperform RNN-based models with baseline activations, exhibiting improved test accuracy.
Paper Structure (16 sections, 7 equations, 9 figures, 3 tables)

This paper contains 16 sections, 7 equations, 9 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Motivation and Intuition behind SST
  4. Squared Sigmoid TanH (SST)
  5. Squared Sigmoid (SS)
  6. Squared TanH (ST)
  7. SST (SS+ST)
  8. Experiments
  9. Sign Language Recognition
  10. Visualization Insights of SST using T-SNE in neural network:
  11. Gait classification
  12. Classification performance:
  13. ROC and AUC curve analysis:
  14. Human activity recognition
  15. Time-series forecasting
  16. ...and 1 more sections

Figures (9)

  • Figure 1: PCA visualization of frames for three distinct ISL classes (a) friend, (b) phone call, and (c) location.
  • Figure 2: SST-AF in a GRU cell.
  • Figure 3: Behaviour of (a) SS and (b) ST.
  • Figure 4: MOPGRU
  • Figure 5: MOPGRU-SST
  • ...and 4 more figures