Motamot: A Dataset for Revealing the Supremacy of Large Language Models over Transformer Models in Bengali Political Sentiment Analysis

TL;DR

This study introduces Motamot, a Bengali political sentiment dataset of 7,058 instances collected from Bangladeshi news outlets to study sentiment toward politics during elections. It systematically compares traditional Pre-trained Language Models (e.g., BanglaBERT, SahajBERT) with Large Language Models (Gemini 1.5 Pro, GPT-3.5 Turbo) using zero-shot and few-shot prompting. The results show PLMs achieve solid baselines (BanglaBERT around 0.881 accuracy), while few-shot LLMs surpass PLMs (Gemini 1.5 Pro up to 0.963 accuracy; GPT-3.5 Turbo up to 0.940), with few-shot prompting reducing hallucinations. The work provides a practical resource for Bengali political sentiment analysis and highlights the potential of few-shot LLM approaches in low-resource languages, offering guidance for researchers and policymakers analyzing online political discourse.

Abstract

Sentiment analysis is the process of identifying and categorizing people's emotions or opinions regarding various topics. Analyzing political sentiment is critical for understanding the complexities of public opinion processes, especially during election seasons. It gives significant information on voter preferences, attitudes, and current trends. In this study, we investigate political sentiment analysis during Bangladeshi elections, specifically examining how effectively Pre-trained Language Models (PLMs) and Large Language Models (LLMs) capture complex sentiment characteristics. Our study centers on the creation of the "Motamot" dataset, comprising 7,058 instances annotated with positive and negative sentiments, sourced from diverse online newspaper portals, forming a comprehensive resource for political sentiment analysis. We meticulously evaluate the performance of various PLMs including BanglaBERT, Bangla BERT Base, XLM-RoBERTa, mBERT, and sahajBERT, alongside LLMs such as Gemini 1.5 Pro and GPT 3.5 Turbo. Moreover, we explore zero-shot and few-shot learning strategies to enhance our understanding of political sentiment analysis methodologies. Our findings underscore BanglaBERT's commendable accuracy of 88.10% among PLMs. However, the exploration into LLMs reveals even more promising results. Through the adept application of Few-Shot learning techniques, Gemini 1.5 Pro achieves an impressive accuracy of 96.33%, surpassing the remarkable performance of GPT 3.5 Turbo, which stands at 94%. This underscores Gemini 1.5 Pro's status as the superior performer in this comparison.

Figures (3)

• Figure 1: This figure provides a visual representation of the Analysis on the Distribution of Political Sentiments Across Train, Test, and Validation Datasets
• Figure 2: An illustration demonstrating the key components of the prompt template designed for zero-shot learning in large language models. The template includes designations for Gemini 1.5 Pro, highlighting System Instruction, Input, and Output, as well as ChatGPT 3.5 Turbo, which highlights System, User, and Assistant interactions
• Figure 3: Visualization of confusion matrices showing the performance of BanglaBERT, Bangla BERT Base, XLM-RoBERTa, mBERT, and sahajBERT pre-trained PLMs in political sentiment analysis. Each subfigure displays the models' classification accuracy across sentiment categories, revealing useful information about their strengths and limitations in sentiment prediction