One Prompt To Rule Them All: LLMs for Opinion Summary Evaluation

TL;DR

To address the poor correlation between reference-based metrics and human judgments in opinion-summaries evaluation, the paper presents SummEval-Op, an 7-dimension benchmark, and two prompt strategies, Op-I-Prompt (dimension-agnostic) and Op-Prompts (dimension-specific), for reference-free evaluation using LLMs. The authors evaluate both closed- and open-source LLMs across 7 dimensions and demonstrate that Op-I-Prompt achieves substantial alignment with human judgments (average Spearman around 0.70), outperforming prior prompts, especially on open-source models. The work includes a new dataset with 32 products from Amazon, 2,912 ratings, and 13 model- and 3 human-generated summaries per product, enabling robust cross-model evaluation. The findings suggest LLM-based, prompt-driven evaluation can surpass traditional metrics for opinion summarization and highlight the need to tailor prompts to model type.

Abstract

Evaluation of opinion summaries using conventional reference-based metrics rarely provides a holistic evaluation and has been shown to have a relatively low correlation with human judgments. Recent studies suggest using Large Language Models (LLMs) as reference-free metrics for NLG evaluation, however, they remain unexplored for opinion summary evaluation. Moreover, limited opinion summary evaluation datasets inhibit progress. To address this, we release the SUMMEVAL-OP dataset covering 7 dimensions related to the evaluation of opinion summaries: fluency, coherence, relevance, faithfulness, aspect coverage, sentiment consistency, and specificity. We investigate Op-I-Prompt a dimension-independent prompt, and Op-Prompts, a dimension-dependent set of prompts for opinion summary evaluation. Experiments indicate that Op-I-Prompt emerges as a good alternative for evaluating opinion summaries achieving an average Spearman correlation of 0.70 with humans, outperforming all previous approaches. To the best of our knowledge, we are the first to investigate LLMs as evaluators on both closed-source and open-source models in the opinion summarization domain.

Figures (5)

• Figure 1: G-Eval vs. Op-I-Prompt. On closed-source model ( ChatGPT-$3.5$) our Op-I-Prompt shows comparable performance whereas on open-source model ( Mistral-$7$B) our approach outperforms G-Eval on $7$ dimensions: fluency (FA), coherence (CO), relevance (RE), faithfulness (FA), aspect coverage (AC), sentiment consistency (SC), and specificity (SP). Check Figure \ref{['fig:generation_runs']} for more details.
• Figure 2: Comparison of Prompt Approaches.G-Eval Prompts first generates the Evaluation Steps using Task Description and Evaluation Criteria in Chain-of-Thought fashion. Finally the full prompt is used to evaluate the opinion summaries. In contrast, our Op-I-Prompt is simpler and has Task Description, Evaluation Criteria, and Evaluation Steps fixed for a dimension/metric independent evaluation. Here, only the Metric part needs to be changed for evaluating any dimension/metric. Finally Op-Prompts are dimension/metric dependent prompts that needs to be specifically crafted for each dimension/metric.
• Figure 3: Ratings Distribution. We plot the average frequency of scores obtained by human raters across $7$ dimensions. A score of $4$ or $5$ is mostly preferred.
• Figure 4: Spearman correlation scores at different number of output generations ( n) for the $7$ dimensions.G-Eval-3.5 and Op-I-GPT-3.5 use the G-Eval and Op-I-Prompt respectively, with closed-source ChatGPT-$3.5$ as their LLM. G-Eval-Mistral, Op-I-Mistral, and Op-Mistral use the G-Eval, Op-I-Prompt, and Op-Prompts respectively, with open-source Mistral-$7$B as their LLM. Generally, Op-I-Prompt shows better relative performance on both closed-source and open-source models.
• Figure 5: Performance of different models as rated by human annotators (Round-II). We observe that GPT-$4$ performs the best followed by Solar-$10.7$B and Mistral-$7$B. Self-supervised models perform worse. In general, all the LLMs perform better than human annotated summaries.