Large Language Models for Sentiment Analysis to Detect Social Challenges: A Use Case with South African Languages

TL;DR

This paper addresses sentiment analysis of social media posts in multilingual South Africa to detect social challenges across government domains. It evaluates zero-shot performance of several state-of-the-art LLMs (GPT-3.5, GPT-4, LLaMa 2, PaLM 2, Dolly 2) on the SAGovTopicTweets corpus in English, Sepedi, and Setswana, and introduces a majority-vote fusion to improve robustness. The results show large differences among models and languages, with fused outputs achieving sentiment classification errors below 1% and strong F1 scores across languages. The work demonstrates the feasibility of language-aware LLM-based sentiment monitoring to inform targeted government actions and highlights considerations such as computational costs and per-topic language variation.

Abstract

Sentiment analysis can aid in understanding people's opinions and emotions on social issues. In multilingual communities sentiment analysis systems can be used to quickly identify social challenges in social media posts, enabling government departments to detect and address these issues more precisely and effectively. Recently, large-language models (LLMs) have become available to the wide public and initial analyses have shown that they exhibit magnificent zero-shot sentiment analysis abilities in English. However, there is no work that has investigated to leverage LLMs for sentiment analysis on social media posts in South African languages and detect social challenges. Consequently, in this work, we analyse the zero-shot performance of the state-of-the-art LLMs GPT-3.5, GPT-4, LlaMa 2, PaLM 2, and Dolly 2 to investigate the sentiment polarities of the 10 most emerging topics in English, Sepedi and Setswana social media posts that fall within the jurisdictional areas of 10 South African government departments. Our results demonstrate that there are big differences between the various LLMs, topics, and languages. In addition, we show that a fusion of the outcomes of different LLMs provides large gains in sentiment classification performance with sentiment classification errors below 1%. Consequently, it is now feasible to provide systems that generate reliable information about sentiment analysis to detect social challenges and draw conclusions about possible needs for actions on specific topics and within different language groups.

Figures (4)

• Figure 1: Pipeline of Topic-Specific Search, Sentiment Analysis and Scoring.
• Figure 2: Zero-Shot Sentiment Classification Workflow with Prompting Example and Expected Response from the LLMs.
• Figure 3: Sentiment distribution of the investigated topics.
• Figure 4: Overall sentiment score per language.