Can Large Language Models be Effective Online Opinion Miners?

TL;DR

The paper tackles the gap between traditional, extraction-centric opinion mining benchmarks and the complex, multi-source online discourse encountered in real settings. It introduces the Online Opinion Mining Benchmark (OOMB), which provides dual-layer annotations (entity-feature-opinion tuples and opinion-centric insights) and two tasks—Feature-centric Opinion Extraction (FOE) and Opinion-centric Insight Generation (OIG)—to assess both structured extraction and abstractive summarization by large language models. Across ten LLMs, results reveal that while LLMs struggle to precisely extract structured opinions, they show relatively stronger performance in generating cohesive, topic-oriented insights, though still exhibit gaps in capturing implicit intents and nuanced contexts. The work establishes a foundation for LLM-based opinion mining in realistic online environments, demonstrates the value of human-in-the-loop annotation, and outlines future directions such as domain generalization, user-aware annotations, and improved evaluation frameworks.

Abstract

The surge of user-generated online content presents a wealth of insights into customer preferences and market trends. However, the highly diverse, complex, and context-rich nature of such contents poses significant challenges to traditional opinion mining approaches. To address this, we introduce Online Opinion Mining Benchmark (OOMB), a novel dataset and evaluation protocol designed to assess the ability of large language models (LLMs) to mine opinions effectively from diverse and intricate online environments. OOMB provides extensive (entity, feature, opinion) tuple annotations and a comprehensive opinion-centric summary that highlights key opinion topics within each content, thereby enabling the evaluation of both the extractive and abstractive capabilities of models. Through our proposed benchmark, we conduct a comprehensive analysis of which aspects remain challenging and where LLMs exhibit adaptability, to explore whether they can effectively serve as opinion miners in realistic online scenarios. This study lays the foundation for LLM-based opinion mining and discusses directions for future research in this field.

Figures (17)

• Figure 1: Existing opinion mining scenarios assume a simple input structure (Upper). In contrast, our study facilitates both extractive and abstractive opinion mining in complex, multi-threaded web discussions, enabling flexible and context-aware mining (Lower).
• Figure 2: The overview of our OOMB benchmark construction pipeline.
• Figure 3: Performance comparison of various LLMs on the FOE task, increasing the numbers of inferences.
• Figure 4: Radar charts for LLM-as-a-judge evaluations of the OIG task. Comparison of the average model performance across different content types (Left). Comparison of performance across different models (Right).
• Figure 5: Performance comparison of various LLMs based on changes in different attributes within online content. CM F1 scores for the FOE task (Upper), and A3CU scores for the OIG task (Lower).