Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Extended LSTM: Adaptive Feature Gating for Toxic Comment Classification

Noor Islam S. Mohammad

TL;DR

Toxic comment detection often faces severe class imbalance and high computational costs when using transformer models. The paper introduces xLSTM, a parameter-efficient architecture that integrates cosine-similarity gating, multi-source embedding fusion, embedding-level SMOTE, and weighted focal loss to concentrate learning on toxic signals while suppressing benign context. Empirically, xLSTM achieves $96.0\%$ accuracy and $0.88$ macro-F1 on the Jigsaw Toxic Comment benchmark, with $15\times$ fewer parameters and $50$ms CPU latency, and shows substantial minority-class improvements (e.g., threat, severe toxic, identity hate) driven by gating. This work demonstrates an efficient, theoretically grounded pathway to outperform large pretrained models on imbalanced, domain-specific NLP tasks, enabling scalable and interpretable deployment in moderation systems.

Abstract

Toxic comment detection remains a challenging task, where transformer-based models (e.g., BERT) incur high computational costs and degrade on minority toxicity classes, while classical ensembles lack semantic adaptability. We propose xLSTM, a parameter-efficient and theoretically grounded framework that unifies cosine-similarity gating, adaptive feature prioritization, and principled class rebalancing. A learnable reference vector {v} in {R}^d modulates contextual embeddings via cosine similarity, amplifying toxic cues and attenuating benign signals to yield stronger gradients under severe class imbalance. xLSTM integrates multi-source embeddings (GloVe, FastText, BERT CLS) through a projection layer, a character-level BiLSTM for morphological cues, embedding-space SMOTE for minority augmentation, and adaptive focal loss with dynamic class weighting. On the Jigsaw Toxic Comment benchmark, xLSTM attains 96.0% accuracy and 0.88 macro-F1, outperforming BERT by 33% on threat and 28% on identity_hate categories, with 15 times fewer parameters and 50ms inference latency. Cosine gating contributes a +4.8% F1 gain in ablations. The results establish a new efficiency adaptability frontier, demonstrating that lightweight, theoretically informed architectures can surpass large pretrained models on imbalanced, domain-specific NLP tasks.

Extended LSTM: Adaptive Feature Gating for Toxic Comment Classification

TL;DR

Toxic comment detection often faces severe class imbalance and high computational costs when using transformer models. The paper introduces xLSTM, a parameter-efficient architecture that integrates cosine-similarity gating, multi-source embedding fusion, embedding-level SMOTE, and weighted focal loss to concentrate learning on toxic signals while suppressing benign context. Empirically, xLSTM achieves accuracy and macro-F1 on the Jigsaw Toxic Comment benchmark, with fewer parameters and ms CPU latency, and shows substantial minority-class improvements (e.g., threat, severe toxic, identity hate) driven by gating. This work demonstrates an efficient, theoretically grounded pathway to outperform large pretrained models on imbalanced, domain-specific NLP tasks, enabling scalable and interpretable deployment in moderation systems.

Abstract

Toxic comment detection remains a challenging task, where transformer-based models (e.g., BERT) incur high computational costs and degrade on minority toxicity classes, while classical ensembles lack semantic adaptability. We propose xLSTM, a parameter-efficient and theoretically grounded framework that unifies cosine-similarity gating, adaptive feature prioritization, and principled class rebalancing. A learnable reference vector {v} in {R}^d modulates contextual embeddings via cosine similarity, amplifying toxic cues and attenuating benign signals to yield stronger gradients under severe class imbalance. xLSTM integrates multi-source embeddings (GloVe, FastText, BERT CLS) through a projection layer, a character-level BiLSTM for morphological cues, embedding-space SMOTE for minority augmentation, and adaptive focal loss with dynamic class weighting. On the Jigsaw Toxic Comment benchmark, xLSTM attains 96.0% accuracy and 0.88 macro-F1, outperforming BERT by 33% on threat and 28% on identity_hate categories, with 15 times fewer parameters and 50ms inference latency. Cosine gating contributes a +4.8% F1 gain in ablations. The results establish a new efficiency adaptability frontier, demonstrating that lightweight, theoretically informed architectures can surpass large pretrained models on imbalanced, domain-specific NLP tasks.
Paper Structure (62 sections, 1 theorem, 39 equations, 9 figures, 6 tables, 2 algorithms)

This paper contains 62 sections, 1 theorem, 39 equations, 9 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Classical and Ensemble Approaches
  4. Deep Learning and Sequential Architectures
  5. Transformer-Based Models and Transfer Learning
  6. Class Imbalance in Neural Networks
  7. Feature Fusion and Multi-Source Embeddings
  8. Cosine-Similarity Gating
  9. Theoretical Analysis and Gradient Signal Concentration
  10. Gradient Concentration Under Extreme Imbalance
  11. Initialization Strategy
  12. Algorithm: Cosine-Similarity Gating Mechanism
  13. Architecture and Components
  14. Multi-Source Embedding Fusion
  15. Theoretic Justification
  16. ...and 47 more sections

Key Result

Theorem 1

Consider a classification model with embedding-based representation $\mathbf{e}_t$. Under position-based attention, the effective learning rate for minority-class features is scaled by the factor $\frac{N_{\text{min}}}{N}$. Under feature-dimension gating with gate $g_t = \sigma(\beta \cdot \text{sim

Figures (9)

  • Figure 1: Multi-task transformer-based architecture for toxic span prediction. The model jointly learns class-level toxicity and token-level toxic spans using co-attention mechanisms.
  • Figure 2: Schematic of the proposed Multichannel Convolutional BiLSTM (MCBLSTM). Each channel applies three Conv1D and max-pooling layers followed by a Bidirectional LSTM. Channel outputs are concatenated and passed through dense layers for toxicity recognition.
  • Figure 3: Neural architecture for text normalization, convolutional feature extraction, and attention-augmented toxicity embedding leading to multilabel classification.
  • Figure 4: Architectural Overview of Memory-Enhanced LSTM Variants. The framework compares four advanced LSTM derivatives—(a) Hypergraph-Memory LSTM (hgLSTM), (b) Neural-Oscillator LSTM (noLSTM), (c) Low-Rank + Sparse Matrix Memory (LRSM), and (d) Contrastive Memory Regularized LSTM (CM). Each variant introduces a distinct inductive bias for structured, rhythmic, or contrastive memory control, aiming to enhance representation fidelity and temporal reasoning in high-dimensional sequential modeling tasks.
  • Figure 5: Architecture of the Proposed Multiview Sequential Forecasting Network (MSFN).
  • ...and 4 more figures

Theorems & Definitions (1)

  • Theorem 1: Gradient Concentration Under Extreme Imbalance