Attention based Bidirectional GRU hybrid model for inappropriate content detection in Urdu language

TL;DR

Urdu inappropriate content detection is challenged by spelling variation and code mixing, exacerbated by resource scarcity. The authors propose an attention based Bidirectional GRU hybrid (BiGRU-A) and benchmark it against LSTM, Bi-LSTM, GRU, and TCN on two Urdu datasets, achieving 84% accuracy on the larger dataset without pre trained Word2Vec embeddings. They show that the attention mechanism improves long range dependency modeling and that pre trained embeddings may not always help due to domain vocabulary gaps. The work also contributes a merged Urdu dataset and insights on dataset size and embedding impact, with future directions including embedding tuning and exploring transformer based approaches.

Abstract

With the increased use of the internet and social networks for online discussions, the spread of toxic and inappropriate content on social networking sites has also increased. Several studies have been conducted in different languages. However, there is less work done for South Asian languages for inappropriate content identification using deep learning techniques. In Urdu language, the spellings are not unique, and people write different common spellings for the same word, while mixing it other languages, like English in the text makes it more challenging, and limited research work is available to process such language with the finest algorithms. The use of attention layer with a deep learning model can help handling the long-term dependencies and increase its efficiency . To explore the effects of the attention layer, this study proposes attention-based Bidirectional GRU hybrid model for identifying inappropriate content in Urdu Unicode text language. Four different baseline deep learning models; LSTM, Bi-LSTM, GRU, and TCN, are used to compare the performance of the proposed model. The results of these models were compared based on evaluation metrics, dataset size, and impact of the word embedding layer. The pre-trained Urdu word2Vec embeddings were utilized for our case. Our proposed model BiGRU-A outperformed all other baseline models by yielding 84\% accuracy without using pre-trained word2Vec layer. From our experiments, we have established that the attention layer improves the model's efficiency, and pre-trained word2Vec embedding does not work well with an inappropriate content dataset.

Figures (5)

• Figure 1: Sample of Dataset.
• Figure 2: Attention-based Bidirectional Gated Recurrent Unit BiGRU - A Proposed model
• Figure 3: Evaluation Metrics Comparison of both data sets with or without Word2Vec
• Figure 4: Accuracy Comparison of both data sets with Word2Vec
• Figure 5: Accuracy Comparison of both data sets without Word2Vec