Decomposed Opinion Summarization with Verified Aspect-Aware Modules

TL;DR

The paper proposes a domain-agnostic modular pipeline for opinion summarization that decomposes the task into Aspect Identification, Opinion Consolidation, and Meta-Review Synthesis, enabling transparent reasoning grounded in review aspects. It leverages zero-shot prompting of large language models to implement each module and process reviews in parallel, across domains including scientific, business, and product reviews. Empirical results show that aspect-aware decomposition improves aspect coverage and opinion faithfulness compared with strong baselines, with Llama-70B excelling at identifying and summarizing aspects and human evaluations indicating higher usefulness of the modular outputs. The work demonstrates that intermediate outputs can assist humans in generating meta-reviews more efficiently and suggests potential applicability to other complex generation tasks.

Abstract

Opinion summarization plays a key role in deriving meaningful insights from large-scale online reviews. To make the process more explainable and grounded, we propose a domain-agnostic modular approach guided by review aspects (e.g., cleanliness for hotel reviews) which separates the tasks of aspect identification, opinion consolidation, and meta-review synthesis to enable greater transparency and ease of inspection. We conduct extensive experiments across datasets representing scientific research, business, and product domains. Results show that our approach generates more grounded summaries compared to strong baseline models, as verified through automated and human evaluations. Additionally, our modular approach, which incorporates reasoning based on review aspects, produces more informative intermediate outputs than other knowledge-agnostic decomposition approaches. Lastly, we provide empirical results to show that these intermediate outputs can support humans in summarizing opinions from large volumes of reviews.

Figures (52)

• Figure 1: High-level overview of our decomposition for opinion summarization using an example from the scientific domain with three aspects (Clarity, Soundness, and Novelty). The modules Aspect Identification, Opinion Consolidation, and Meta-Review Synthesis are instantiated with prompt-based LLMs and operate in sequence. The output of Aspect Identification serves as input to Opinion consolidation and Meta-Review synthesis aggregates opinions found in aspect-specific meta-reviews. All prompts and inputs/outputs are in natural language.
• Figure 2: Evaluation of text fragments extracted for individual review aspects by Aspect Identification.
• Figure 3: The few-shot prompt template for the Aspect Identification module; text fragments are extracted for each (domain) aspect. Please note that for research articles we use few-shot prompting to enable the model follow the output format while for sports shoes and hotels zero-shot prompting (with just removing the demonstration example) could get reasonable performances.
• Figure 4: The few-shot prompt template for the Opinion Consolidation module; it outputs summaries for individual review aspects. Please note that for research articles we use few-shot prompting to get better performance while for sports shoes and hotels zero-shot prompting (with just removing the demonstration example) could get reasonable performances.
• Figure 5: The prompt template for the Meta-Review Synthesis module based on aspect-specific meta-reviews from the Opinion Consolidation module. As zero-shot prompting gives us reasonable performances on all the three datasets, we used the same zero-shot prompt template for the module.