Enhancing Granular Sentiment Classification with Chain-of-Thought Prompting in Large Language Models

TL;DR

The paper addresses the challenge of extracting fine-grained sentiment from app reviews, where traditional polarity or star ratings fail to reflect nuanced experiences. It implements Chain-of-Thought prompting with GPT-4 to induce explicit reasoning for sentiment tagging and benchmarks it against a simple prompt approach on 2,000 Amazon App Store reviews. Results show CoT prompting improves accuracy from 84% to 93% and reduces a large share of errors caused by mixed or ambiguous sentiment, demonstrating the value of structured reasoning for real-world text. The work offers practical implications for sentiment-aware feedback analysis and points to future work in multilingual and multimodal extensions and potential integration with fine-tuning for further gains.

Abstract

We explore the use of Chain-of-Thought (CoT) prompting with large language models (LLMs) to improve the accuracy of granular sentiment categorization in app store reviews. Traditional numeric and polarity-based ratings often fail to capture the nuanced sentiment embedded in user feedback. We evaluated the effectiveness of CoT prompting versus simple prompting on 2000 Amazon app reviews by comparing each method's predictions to human judgements. CoT prompting improved classification accuracy from 84% to 93% highlighting the benefit of explicit reasoning in enhancing sentiment analysis performance.