AtteSTNet -- An attention and subword tokenization based approach for code-switched text hate speech detection

TL;DR

This work tackles Hinglish code-switched hate speech detection by introducing AtteSTNet, a lightweight, non-recurrent model that combines dual subword tokenization (BPE and Unigram) with multi-head self-attention and fixed positional encodings. By processing two encoded streams (one pre-padded for BPE and one post-padded for Unigram) and concatenating their representations before classification, the approach achieves a strong 87.41% accuracy and 0.851 F1 on standard Hinglish datasets, without relying on profanity lexicons or translation dictionaries. The method offers comparable or better performance than more complex recurrent or convolutional models while reducing computational requirements, highlighting the practicality of attention-based architectures for code-switched language. The work also emphasizes the effectiveness of subword tokenization in handling non-conventional Hinglish vocabulary and demonstrates the method’s potential applicability to other code-switched languages in real-world deployments.

Abstract

Recent advancements in technology have led to a boost in social media usage which has ultimately led to large amounts of user-generated data which also includes hateful and offensive speech. The language used in social media is often a combination of English and the native language in the region. In India, Hindi is used predominantly and is often code-switched with English, giving rise to the Hinglish (Hindi+English) language. Various approaches have been made in the past to classify the code-mixed Hinglish hate speech using different machine learning and deep learning-based techniques. However, these techniques make use of recurrence on convolution mechanisms which are computationally expensive and have high memory requirements. Past techniques also make use of complex data processing making the existing techniques very complex and non-sustainable to change in data. Proposed work gives a much simpler approach which is not only at par with these complex networks but also exceeds performance with the use of subword tokenization algorithms like BPE and Unigram, along with multi-head attention-based techniques, giving an accuracy of 87.41% and an F1 score of 0.851 on standard datasets. Efficient use of BPE and Unigram algorithms help handle the nonconventional Hinglish vocabulary making the proposed technique simple, efficient and sustainable to use in the real world.

Figures (3)

• Figure 1: Block diagram of the proposed model architecture
• Figure 2: Schematic representation of self dot product attention attention
• Figure 3: Multi head attention consisting of parallel running attention layers attention