On the Impact of Language Nuances on Sentiment Analysis with Large Language Models: Paraphrasing, Sarcasm, and Emojis

TL;DR

This study investigates how language nuances—sarcasm, emojis, and fragmented text—affect sentiment analysis with large language models. It evaluates data-quality interventions, including a human-labeled sarcasm dataset, text paraphrasing, and adversarial text augmentation, across domain-specific (nuclear power) and general tweet datasets. Key findings show that domain-specific models are notably sensitive to sarcasm and benefit from sarcasm removal or augmentation, while general data improves sarcasm robustness; paraphrasing enhances performance on low-quality text, and adversarial augmentation yields strong gains, with emojis offering limited additional value in this context. The work highlights data diversity and quality as critical levers for reliable LLM-based sentiment analysis on social media.

Abstract

Large Language Models (LLMs) have demonstrated impressive performance across various tasks, including sentiment analysis. However, data quality--particularly when sourced from social media--can significantly impact their accuracy. This research explores how textual nuances, including emojis and sarcasm, affect sentiment analysis, with a particular focus on improving data quality through text paraphrasing techniques. To address the lack of labeled sarcasm data, the authors created a human-labeled dataset of 5929 tweets that enabled the assessment of LLM in various sarcasm contexts. The results show that when topic-specific datasets, such as those related to nuclear power, are used to finetune LLMs these models are not able to comprehend accurate sentiment in presence of sarcasm due to less diverse text, requiring external interventions like sarcasm removal to boost model accuracy. Sarcasm removal led to up to 21% improvement in sentiment accuracy, as LLMs trained on nuclear power-related content struggled with sarcastic tweets, achieving only 30% accuracy. In contrast, LLMs trained on general tweet datasets, covering a broader range of topics, showed considerable improvements in predicting sentiment for sarcastic tweets (60% accuracy), indicating that incorporating general text data can enhance sarcasm detection. The study also utilized adversarial text augmentation, showing that creating synthetic text variants by making minor changes significantly increased model robustness and accuracy for sarcastic tweets (approximately 85%). Additionally, text paraphrasing of tweets with fragmented language transformed around 40% of the tweets with low-confidence labels into high-confidence ones, improving LLMs sentiment analysis accuracy by 6%.

Figures (5)

• Figure 1: Distribution of sarcasm labels in the human-labelled dataset
• Figure 2: Sentiment distribution for the human-labeled dataset used for sarcasm analysis
• Figure 3: Llama-2 confusion matrix when fine-tuned on the paraphrased nuclear power tweets.
• Figure 4: Library Agreement distribution of original vs paraphrased tweets.
• Figure 5: Confusion Matrices of the Falcon model finetuned on Nuclear Power data.