Dynamic Sentiment Analysis with Local Large Language Models using Majority Voting: A Study on Factors Affecting Restaurant Evaluation
Junichiro Niimi
TL;DR
The paper addresses robust, privacy-preserving sentiment analysis of user-generated reviews by leveraging local, quantized LLMs for aspect-based sentiment analysis. It introduces a two-stage majority voting mechanism that aggregates outputs from five virtual workers to improve robustness and reproducibility, showing that a medium-sized Llama model with 4-bit quantization can outperform a large single model in both accuracy and efficiency. Through experiments on Yelp restaurant reviews, it demonstrates that majority voting yields significant accuracy gains and that predicted aspect sentiments align with actual ratings in downstream regression analyses, validating generalizability. The approach offers practical benefits for marketing research by enabling on-device analysis with reduced security risk and controllable costs, while outlining limitations and avenues for future work.
Abstract
User-generated contents (UGCs) on online platforms allow marketing researchers to understand consumer preferences for products and services. With the advance of large language models (LLMs), some studies utilized the models for annotation and sentiment analysis. However, the relationship between the accuracy and the hyper-parameters of LLMs is yet to be thoroughly examined. In addition, the issues of variability and reproducibility of results from each trial of LLMs have rarely been considered in existing literature. Since actual human annotation uses majority voting to resolve disagreements among annotators, this study introduces a majority voting mechanism to a sentiment analysis model using local LLMs. By a series of three analyses of online reviews on restaurant evaluations, we demonstrate that majority voting with multiple attempts using a medium-sized model produces more robust results than using a large model with a single attempt. Furthermore, we conducted further analysis to investigate the effect of each aspect on the overall evaluation.