Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Domain-Expanded ASTE: Rethinking Generalization in Aspect Sentiment Triplet Extraction

Yew Ken Chia, Hui Chen, Wei Han, Guizhen Chen, Sharifah Mahani Aljunied, Soujanya Poria, Lidong Bing

TL;DR

This work introduces a domain-expanded benchmark by annotating samples from diverse domains, enabling evaluation of models in both in-domain and out-of-domain settings, and proposes CASE, a simple and effective decoding strategy that enhances trustworthiness and performance of LLMs in ASTE.

Abstract

Aspect Sentiment Triplet Extraction (ASTE) is a challenging task in sentiment analysis, aiming to provide fine-grained insights into human sentiments. However, existing benchmarks are limited to two domains and do not evaluate model performance on unseen domains, raising concerns about the generalization of proposed methods. Furthermore, it remains unclear if large language models (LLMs) can effectively handle complex sentiment tasks like ASTE. In this work, we address the issue of generalization in ASTE from both a benchmarking and modeling perspective. We introduce a domain-expanded benchmark by annotating samples from diverse domains, enabling evaluation of models in both in-domain and out-of-domain settings. Additionally, we propose CASE, a simple and effective decoding strategy that enhances trustworthiness and performance of LLMs in ASTE. Through comprehensive experiments involving multiple tasks, settings, and models, we demonstrate that CASE can serve as a general decoding strategy for complex sentiment tasks. By expanding the scope of evaluation and providing a more reliable decoding strategy, we aim to inspire the research community to reevaluate the generalizability of benchmarks and models for ASTE. Our code, data, and models are available at https://github.com/DAMO-NLP-SG/domain-expanded-aste.

Domain-Expanded ASTE: Rethinking Generalization in Aspect Sentiment Triplet Extraction

TL;DR

This work introduces a domain-expanded benchmark by annotating samples from diverse domains, enabling evaluation of models in both in-domain and out-of-domain settings, and proposes CASE, a simple and effective decoding strategy that enhances trustworthiness and performance of LLMs in ASTE.

Abstract

Aspect Sentiment Triplet Extraction (ASTE) is a challenging task in sentiment analysis, aiming to provide fine-grained insights into human sentiments. However, existing benchmarks are limited to two domains and do not evaluate model performance on unseen domains, raising concerns about the generalization of proposed methods. Furthermore, it remains unclear if large language models (LLMs) can effectively handle complex sentiment tasks like ASTE. In this work, we address the issue of generalization in ASTE from both a benchmarking and modeling perspective. We introduce a domain-expanded benchmark by annotating samples from diverse domains, enabling evaluation of models in both in-domain and out-of-domain settings. Additionally, we propose CASE, a simple and effective decoding strategy that enhances trustworthiness and performance of LLMs in ASTE. Through comprehensive experiments involving multiple tasks, settings, and models, we demonstrate that CASE can serve as a general decoding strategy for complex sentiment tasks. By expanding the scope of evaluation and providing a more reliable decoding strategy, we aim to inspire the research community to reevaluate the generalizability of benchmarks and models for ASTE. Our code, data, and models are available at https://github.com/DAMO-NLP-SG/domain-expanded-aste.
Paper Structure (37 sections, 6 equations, 3 figures, 10 tables, 1 algorithm)

This paper contains 37 sections, 6 equations, 3 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Aspect-Based Sentiment Analysis
  4. Domain Generalization
  5. Large Language Models
  6. Domain-Expanded ASTE Benchmark
  7. Task Formulation
  8. Data Collection
  9. Data Annotation
  10. Confidence-Aware Sentiment Extraction (CASE)
  11. Candidate Sampling
  12. Aggregation
  13. Confidence Estimation
  14. Filtering
  15. Experiment Setup
  16. ...and 22 more sections

Figures (3)

  • Figure 1: ASTE data samples for the Hotel, Laptop, Cosmetics, and Restaurant domains, respectively.
  • Figure 2: Our proposed confidence-aware sentiment extraction (CASE) decoding strategy which aims to enhance the trustworthiness and performance of LLMs for ASTE.
  • Figure 3: The effect of confidence-aware threshold $T$ on in-domain performance for the Hotel domain.