Single Ground Truth Is Not Enough: Adding Flexibility to Aspect-Based Sentiment Analysis Evaluation

TL;DR

This work tackles the misalignment between ABSA ground-truth singletons and the diverse surface forms naturally expressed in language. It introduces Zoom In-N-Out, a fully automated GT-expansion pipeline that refines, extends, and validates aspect and opinion terms to produce $Y^{\text{new}}$ for fairer evaluation. Empirical results show higher human agreement and substantial gains for LLMs over traditional GTs (up to roughly 10 percentage points in Kendall's Tau and large F1 uplifts for LLMs), revealing latent ABSA capabilities previously masked by exact-match GTs. The method, cost-effective and reproducible, suggests a paradigm shift in evaluating span-extraction models and generalizes to other NLP tasks involving surface-form variability.

Abstract

Aspect-based sentiment analysis (ABSA) is a challenging task of extracting sentiments along with their corresponding aspects and opinion terms from the text. The inherent subjectivity of span annotation makes variability in the surface forms of extracted terms, complicating the evaluation process. Traditional evaluation methods often constrain ground truths (GT) to a single term, potentially misrepresenting the accuracy of semantically valid predictions that differ in surface form. To address this limitation, we propose a novel and fully automated pipeline that expands existing evaluation sets by adding alternative valid terms for aspect and opinion. Our approach facilitates an equitable assessment of language models by accommodating multiple-answer candidates, resulting in enhanced human agreement compared to single-answer test sets (achieving up to a 10\%p improvement in Kendall's Tau score). Experimental results demonstrate that our expanded evaluation set helps uncover the capabilities of large language models (LLMs) in ABSA tasks, which is concealed by the single-answer GT sets. Consequently, our work contributes to the development of a flexible evaluation framework for ABSA by embracing diverse surface forms to span extraction tasks in a cost-effective and reproducible manner. Our code and dataset is open at https://github.com/dudrrm/zoom-in-n-out-absa.

Figures (5)

• Figure 1: Original ground truth (GT) aspect and opinion terms are shown in gray background shading. Alternative valid expressions for these terms are highlighted: aspects in yellow and opinions in blue. Under conventional evaluation approaches using the original GT set, these highlighted candidates are assessed as wrong because they do not match exactly despite being semantically consistent with the original terms.
• Figure 2: Overview of Zoom In-N-Out, illustrating the extraction of diverse candidates of the aspect and opinion terms in ABSA task. The three stages of our pipeline are depicted by three colored blocks. Starting with the input sentence and the original ground truth term highlighted in yellow, the model generates alternative representations of the target term at each stage. The original ground truth quadruple is ("9 oz steak", Food Quality, Negative, "n't worth"), representing aspect, category, sentiment, and opinion, respectively. Since our pipeline preserves the provided categorical values, the category and sentiment are not included in the figure.
• Figure 3: Average and standard deviation of word counts in explicit aspect and opinion terms across the four datasets.
• Figure 4: Dataset validity study UI example on ACOS-Rest dataset in Section \ref{['sec:dataset-validity']}.
• Figure 5: Human evaluation study UI example on ACOS-Rest dataset.