A hybrid transformer and attention based recurrent neural network for robust and interpretable sentiment analysis of tweets

TL;DR

The TRABSA model outperforms the current seven traditional machine learning models, four stacking models, and four hybrid deep learning models, yielding notable gain in accuracy and effectiveness with a macro average precision of 94%, recall of 93%, and F1-score of 94%.

Abstract

Sentiment analysis is crucial for understanding public opinion and consumer behavior. Existing models face challenges with linguistic diversity, generalizability, and explainability. We propose TRABSA, a hybrid framework integrating transformer-based architectures, attention mechanisms, and BiLSTM networks to address this. Leveraging RoBERTa-trained on 124M tweets, we bridge gaps in sentiment analysis benchmarks, ensuring state-of-the-art accuracy. Augmenting datasets with tweets from 32 countries and US states, we compare six word-embedding techniques and three lexicon-based labeling techniques, selecting the best for optimal sentiment analysis. TRABSA outperforms traditional ML and deep learning models with 94% accuracy and significant precision, recall, and F1-score gains. Evaluation across diverse datasets demonstrates consistent superiority and generalizability. SHAP and LIME analyses enhance interpretability, improving confidence in predictions. Our study facilitates pandemic resource management, aiding resource planning, policy formation, and vaccination tactics.

Figures (9)

• Figure 1: This figure depicts the step-by-step methodological framework proposed for tweet sentiment analysis. It begins with (a) data collection and extension, followed by (b) data cleaning and preprocessing. Subsequently, (c) sentiment labeling into positive, neutral, and negative categories is performed using the 'cardiffnlp/twitter-roberta-base-sentiment-latest' pre-trained transformer, and the dataset is split into training, validation, and test sets. The framework proceeds with (d) model development, (e) model benchmarking, and evaluation against baseline models or state-of-the-art approaches. Finally, the process concludes with XAI interpretation techniques applied to gain insights into the model's predictions.
• Figure 2: Stacked bar chart showing tweet distribution in three stages of data collection during third lockdown period from the major cities of the UK.
• Figure 4: This word cloud visualizes the most frequently occurring words in the "Global" (left) and "Only USA" (right) datasets of COVID-19-related tweets. The size of each word corresponds to its frequency in the dataset.
• Figure 5: Model architecture of the proposed TRABSA model.
• Figure 6: This figure depicts the learning rate scheduler callback function utilized during model training. It visualizes the decayed learning rate as the epoch increases, alongside the corresponding training and validation accuracy, as well as training and validation loss, plotted against variable learning rates.