Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Comparing Without Saying: A Dataset and Benchmark for Implicit Comparative Opinion Mining from Same-User Reviews

Thanh-Lam T. Nguyen, Ngoc-Quang Le, Quoc-Trung Phu, Thi-Phuong Le, Ngoc-Huyen Pham, Phuong-Nguyen Nguyen, Hoang-Quynh Le

TL;DR

This work tackles implicit comparative opinion mining by introducing SUDO, a bi-level annotated dataset of same-user beer reviews capturing aspect mentions and per-aspect comparative preferences. It combines a two-stage annotation pipeline with rigorous data filtering and a user-level train/validation/test split to enable robust evaluation. Through comprehensive benchmarks, traditional ML baselines underperform compared with end-to-end language-model baselines, though overall accuracy remains moderate due to implicit reasoning and cross-aspect interactions. By providing a large, publicly available benchmark focused on implicit comparisons, the paper highlights key challenges and motivates future work in joint learning and cross-domain expansion.

Abstract

Existing studies on comparative opinion mining have mainly focused on explicit comparative expressions, which are uncommon in real-world reviews. This leaves implicit comparisons - here users express preferences across separate reviews - largely underexplored. We introduce SUDO, a novel dataset for implicit comparative opinion mining from same-user reviews, allowing reliable inference of user preferences even without explicit comparative cues. SUDO comprises 4,150 annotated review pairs (15,191 sentences) with a bi-level structure capturing aspect-level mentions and review-level preferences. We benchmark this task using two baseline architectures: traditional machine learning- and language model-based baselines. Experimental results show that while the latter outperforms the former, overall performance remains moderate, revealing the inherent difficulty of the task and establishing SUDO as a challenging and valuable benchmark for future research.

Comparing Without Saying: A Dataset and Benchmark for Implicit Comparative Opinion Mining from Same-User Reviews

TL;DR

This work tackles implicit comparative opinion mining by introducing SUDO, a bi-level annotated dataset of same-user beer reviews capturing aspect mentions and per-aspect comparative preferences. It combines a two-stage annotation pipeline with rigorous data filtering and a user-level train/validation/test split to enable robust evaluation. Through comprehensive benchmarks, traditional ML baselines underperform compared with end-to-end language-model baselines, though overall accuracy remains moderate due to implicit reasoning and cross-aspect interactions. By providing a large, publicly available benchmark focused on implicit comparisons, the paper highlights key challenges and motivates future work in joint learning and cross-domain expansion.

Abstract

Existing studies on comparative opinion mining have mainly focused on explicit comparative expressions, which are uncommon in real-world reviews. This leaves implicit comparisons - here users express preferences across separate reviews - largely underexplored. We introduce SUDO, a novel dataset for implicit comparative opinion mining from same-user reviews, allowing reliable inference of user preferences even without explicit comparative cues. SUDO comprises 4,150 annotated review pairs (15,191 sentences) with a bi-level structure capturing aspect-level mentions and review-level preferences. We benchmark this task using two baseline architectures: traditional machine learning- and language model-based baselines. Experimental results show that while the latter outperforms the former, overall performance remains moderate, revealing the inherent difficulty of the task and establishing SUDO as a challenging and valuable benchmark for future research.
Paper Structure (46 sections, 11 equations, 11 figures, 7 tables)

This paper contains 46 sections, 11 equations, 11 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset Construction
  4. Annotation Pipeline
  5. Data Filtering and Preprocessing
  6. Data Filtering
  7. Preprocessing
  8. Data Annotation
  9. Training Annotators
  10. Bi-level Data Annotation
  11. Data Forming
  12. Comparisons With Other Datasets
  13. Data Statistics and Analysis
  14. Label Distribution
  15. Cross-Aspects Relationship Analysis
  16. ...and 31 more sections

Figures (11)

  • Figure 1: Illustration of explicit vs. implicit comparative opinions in user reviews.
  • Figure 2: Data annotation pipeline.
  • Figure 3: Distribution of aspect and comparative labels.
  • Figure 4: Statistics on lexical overlap between some aspect pairs.
  • Figure 5: Aspect co-occurrence within a sentence
  • ...and 6 more figures