Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ChatASU: Evoking LLM's Reflexion to Truly Understand Aspect Sentiment in Dialogues

Yiding Liu, Jingjing Wang, Jiamin Luo, Tao Zeng, Guodong Zhou

TL;DR

A Trusted Self-reflexion Approach (TSA) with ChatGLM as backbone to ChatASU, aiming to explore LLMs’ ability in understanding aspect sentiments in dialogue scenarios and justifying the effectiveness of TSA to ChatASU and the importance of considering the coreference and hallucination issues in ChatASU.

Abstract

Aspect Sentiment Understanding (ASU) in interactive scenarios (e.g., Question-Answering and Dialogue) has attracted ever-more interest in recent years and achieved important progresses. However, existing studies on interactive ASU largely ignore the coreference issue for opinion targets (i.e., aspects), while this phenomenon is ubiquitous in interactive scenarios especially dialogues, limiting the ASU performance. Recently, large language models (LLMs) shows the powerful ability to integrate various NLP tasks with the chat paradigm. In this way, this paper proposes a new Chat-based Aspect Sentiment Understanding (ChatASU) task, aiming to explore LLMs' ability in understanding aspect sentiments in dialogue scenarios. Particularly, this ChatASU task introduces a sub-task, i.e., Aspect Chain Reasoning (ACR) task, to address the aspect coreference issue. On this basis, we propose a Trusted Self-reflexion Approach (TSA) with ChatGLM as backbone to ChatASU. Specifically, this TSA treats the ACR task as an auxiliary task to boost the performance of the primary ASU task, and further integrates trusted learning into reflexion mechanisms to alleviate the LLMs-intrinsic factual hallucination problem in TSA. Furthermore, a high-quality ChatASU dataset is annotated to evaluate TSA, and extensive experiments show that our proposed TSA can significantly outperform several state-of-the-art baselines, justifying the effectiveness of TSA to ChatASU and the importance of considering the coreference and hallucination issues in ChatASU.

ChatASU: Evoking LLM's Reflexion to Truly Understand Aspect Sentiment in Dialogues

TL;DR

A Trusted Self-reflexion Approach (TSA) with ChatGLM as backbone to ChatASU, aiming to explore LLMs’ ability in understanding aspect sentiments in dialogue scenarios and justifying the effectiveness of TSA to ChatASU and the importance of considering the coreference and hallucination issues in ChatASU.

Abstract

Aspect Sentiment Understanding (ASU) in interactive scenarios (e.g., Question-Answering and Dialogue) has attracted ever-more interest in recent years and achieved important progresses. However, existing studies on interactive ASU largely ignore the coreference issue for opinion targets (i.e., aspects), while this phenomenon is ubiquitous in interactive scenarios especially dialogues, limiting the ASU performance. Recently, large language models (LLMs) shows the powerful ability to integrate various NLP tasks with the chat paradigm. In this way, this paper proposes a new Chat-based Aspect Sentiment Understanding (ChatASU) task, aiming to explore LLMs' ability in understanding aspect sentiments in dialogue scenarios. Particularly, this ChatASU task introduces a sub-task, i.e., Aspect Chain Reasoning (ACR) task, to address the aspect coreference issue. On this basis, we propose a Trusted Self-reflexion Approach (TSA) with ChatGLM as backbone to ChatASU. Specifically, this TSA treats the ACR task as an auxiliary task to boost the performance of the primary ASU task, and further integrates trusted learning into reflexion mechanisms to alleviate the LLMs-intrinsic factual hallucination problem in TSA. Furthermore, a high-quality ChatASU dataset is annotated to evaluate TSA, and extensive experiments show that our proposed TSA can significantly outperform several state-of-the-art baselines, justifying the effectiveness of TSA to ChatASU and the importance of considering the coreference and hallucination issues in ChatASU.
Paper Structure (18 sections, 12 equations, 4 figures, 2 tables)

This paper contains 18 sections, 12 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Aspect Sentiment Understanding
  4. Reflexion Mechanism
  5. Dataset Construction for ChatASU
  6. Trusted Self-reflexion Approach
  7. ChatASU Block
  8. Trusted Enhanced Self-reflexion
  9. Optimization for TSA
  10. Experiments
  11. Experimental Settings and Metrics
  12. Baselines
  13. Experimental Results
  14. Analysis and Discussions
  15. Importance and Robustness Study for ACR Task and Trusted Learning
  16. ...and 3 more sections

Figures (4)

  • Figure 1: An example to illustrate the coreference and hallucination issue. (A): The concrete dialogue to explain the proposed Aspect Chain and Hallucinations in ChatASU, where different colors represent different aspects. (B): Two Aspect Chains of "Wen Chaorong" and "Zhang Zhongwei" with there corresponding coreference in this dialogue, where NoCoreference means that the current utterance has no coreference. (C): The factual hallucinations exist in ChatASU, i.e., errors in extracting the coreference and predicting the sentiment.
  • Figure 2: The overall framework of our Trusted Self-reflexion Approach (TSA), consisting of ChatASU Block and Trusted Enhanced Self-reflexion (TES) Block.
  • Figure 3: (a) The performance of our TSA to ACR task with or without TL during different training steps. (b) The performance of our TSA to ASU task with or without TL during different training steps.
  • Figure 4: A dialogue example (eight utterances) with their four ground-truth quadruple from the test data of ChatASU dataset. Normalized Predicted Scores denote the normalized predicted percentage for top-3 and "other" generated answers. GT denotes the ground-truth.