Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Fusion of context-aware based BanglaBERT and Two-Layer Stacked LSTM Framework for Multi-Label Cyberbullying Detection

Mirza Raquib, Asif Pervez Polok, Kedar Nath Biswas, Rahat Uddin Azad, Saydul Akbar Murad, Nick Rahimi

TL;DR

A fusion architecture that combines BanglaBERT-Large with a two-layer stacked LSTM with different sampling strategies to address class imbalance is proposed and evaluated on a publicly available multilabel Bangla cyberbullying dataset.

Abstract

Cyberbullying has become a serious and growing concern in todays virtual world. When left unnoticed, it can have adverse consequences for social and mental health. Researchers have explored various types of cyberbullying, but most approaches use single-label classification, assuming that each comment contains only one type of abuse. In reality, a single comment may include overlapping forms such as threats, hate speech, and harassment. Therefore, multilabel detection is both realistic and essential. However, multilabel cyberbullying detection has received limited attention, especially in low-resource languages like Bangla, where robust pre-trained models are scarce. Developing a generalized model with moderate accuracy remains challenging. Transformers offer strong contextual understanding but may miss sequential dependencies, while LSTM models capture temporal flow but lack semantic depth. To address these limitations, we propose a fusion architecture that combines BanglaBERT-Large with a two-layer stacked LSTM. We analyze their behavior to jointly model context and sequence. The model is fine-tuned and evaluated on a publicly available multilabel Bangla cyberbullying dataset covering cyberbully, sexual harassment, threat, and spam. We apply different sampling strategies to address class imbalance. Evaluation uses multiple metrics, including accuracy, precision, recall, F1-score, Hamming loss, Cohens kappa, and AUC-ROC. We employ 5-fold cross-validation to assess the generalization of the architecture.

A Fusion of context-aware based BanglaBERT and Two-Layer Stacked LSTM Framework for Multi-Label Cyberbullying Detection

TL;DR

A fusion architecture that combines BanglaBERT-Large with a two-layer stacked LSTM with different sampling strategies to address class imbalance is proposed and evaluated on a publicly available multilabel Bangla cyberbullying dataset.

Abstract

Cyberbullying has become a serious and growing concern in todays virtual world. When left unnoticed, it can have adverse consequences for social and mental health. Researchers have explored various types of cyberbullying, but most approaches use single-label classification, assuming that each comment contains only one type of abuse. In reality, a single comment may include overlapping forms such as threats, hate speech, and harassment. Therefore, multilabel detection is both realistic and essential. However, multilabel cyberbullying detection has received limited attention, especially in low-resource languages like Bangla, where robust pre-trained models are scarce. Developing a generalized model with moderate accuracy remains challenging. Transformers offer strong contextual understanding but may miss sequential dependencies, while LSTM models capture temporal flow but lack semantic depth. To address these limitations, we propose a fusion architecture that combines BanglaBERT-Large with a two-layer stacked LSTM. We analyze their behavior to jointly model context and sequence. The model is fine-tuned and evaluated on a publicly available multilabel Bangla cyberbullying dataset covering cyberbully, sexual harassment, threat, and spam. We apply different sampling strategies to address class imbalance. Evaluation uses multiple metrics, including accuracy, precision, recall, F1-score, Hamming loss, Cohens kappa, and AUC-ROC. We employ 5-fold cross-validation to assess the generalization of the architecture.
Paper Structure (35 sections, 34 equations, 3 figures, 9 tables, 3 algorithms)

This paper contains 35 sections, 34 equations, 3 figures, 9 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Methodology
  4. Dataset Collection
  5. Undersampling and Oversampling
  6. Data Preprocessing
  7. BERT Tokenizer
  8. Padding and Truncation.
  9. Attention Mask Generation.
  10. Proposed Model Architecture Overview
  11. BanglaBERT-Large Layer
  12. Input Embedding Layer
  13. Transformer Encoder Layers
  14. LSTM Layers
  15. LSTM Gate Computation
  16. ...and 20 more sections

Figures (3)

  • Figure 1: Workflow of the Proposed Cyberbullying Detection
  • Figure 2: Model Architecture of Proposed BERT-CNN-BiLSTM
  • Figure 3: Multi-label Classification Predictions Showing Class Probabilities and Word-level Importance Scores for Each Cyberbullying Category. The Highlighted Words in the Bangla Text Indicate Their Contribution to the Model's Prediction for Each Label.