Improving In-Context Learning with Prediction Feedback for Sentiment Analysis

TL;DR

The paper addresses the limited ability of in-context learning by LLMs to discern subtle sentiments. It introduces a prediction-feedback framework that first extracts prior predictions, then provides correctness-based and analysis-based feedback to form a feedback-driven prompt for test-time inference. Across nine sentiment datasets, the method yields statistically meaningful F1 improvements and demonstrates robustness across LLMs and tasks, including stance, irony, and NLI. The approach emphasizes retrieval-agnostic prompt construction and human-aligned reasoning, offering practical improvements for resource-constrained settings and broader applicability to NLP tasks.

Abstract

Large language models (LLMs) have achieved promising results in sentiment analysis through the in-context learning (ICL) paradigm. However, their ability to distinguish subtle sentiments still remains a challenge. Inspired by the human ability to adjust understanding via feedback, this paper enhances ICL by incorporating prior predictions and feedback, aiming to rectify sentiment misinterpretation of LLMs. Specifically, the proposed framework consists of three steps: (1) acquiring prior predictions of LLMs, (2) devising predictive feedback based on correctness, and (3) leveraging a feedback-driven prompt to refine sentiment understanding. Experimental results across nine sentiment analysis datasets demonstrate the superiority of our framework over conventional ICL methods, with an average F1 improvement of 5.95%.

Figures (6)

• Figure 1: Normalized confusion matrices on two sentiment analysis datasets. Results are from ChatGPT.
• Figure 2: Overview of our framework.
• Figure 3: Normalized confusion matrices for the Poem dataset: K-Means (left) and K-Means+Ours (right). See results for more datasets in Appendix \ref{['sec: more_effect']}.
• Figure 4: Impact of the error example ratio.
• Figure 5: Effect on subtle sentiments for other datasets.