Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ORCHID: A Chinese Debate Corpus for Target-Independent Stance Detection and Argumentative Dialogue Summarization

Xiutian Zhao, Ke Wang, Wei Peng

TL;DR

This work presents ORCHID (Oral Chinese Debate), the first Chinese dataset for benchmarking target-independent stance detection and debate summarization, and suggests a potential of incorporating stance detection in summarization for argumentative dialogue.

Abstract

Dialogue agents have been receiving increasing attention for years, and this trend has been further boosted by the recent progress of large language models (LLMs). Stance detection and dialogue summarization are two core tasks of dialogue agents in application scenarios that involve argumentative dialogues. However, research on these tasks is limited by the insufficiency of public datasets, especially for non-English languages. To address this language resource gap in Chinese, we present ORCHID (Oral Chinese Debate), the first Chinese dataset for benchmarking target-independent stance detection and debate summarization. Our dataset consists of 1,218 real-world debates that were conducted in Chinese on 476 unique topics, containing 2,436 stance-specific summaries and 14,133 fully annotated utterances. Besides providing a versatile testbed for future research, we also conduct an empirical study on the dataset and propose an integrated task. The results show the challenging nature of the dataset and suggest a potential of incorporating stance detection in summarization for argumentative dialogue.

ORCHID: A Chinese Debate Corpus for Target-Independent Stance Detection and Argumentative Dialogue Summarization

TL;DR

This work presents ORCHID (Oral Chinese Debate), the first Chinese dataset for benchmarking target-independent stance detection and debate summarization, and suggests a potential of incorporating stance detection in summarization for argumentative dialogue.

Abstract

Dialogue agents have been receiving increasing attention for years, and this trend has been further boosted by the recent progress of large language models (LLMs). Stance detection and dialogue summarization are two core tasks of dialogue agents in application scenarios that involve argumentative dialogues. However, research on these tasks is limited by the insufficiency of public datasets, especially for non-English languages. To address this language resource gap in Chinese, we present ORCHID (Oral Chinese Debate), the first Chinese dataset for benchmarking target-independent stance detection and debate summarization. Our dataset consists of 1,218 real-world debates that were conducted in Chinese on 476 unique topics, containing 2,436 stance-specific summaries and 14,133 fully annotated utterances. Besides providing a versatile testbed for future research, we also conduct an empirical study on the dataset and propose an integrated task. The results show the challenging nature of the dataset and suggest a potential of incorporating stance detection in summarization for argumentative dialogue.

Paper Structure

This paper contains 41 sections, 3 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work: Existing Datasets
  3. Stance Detection Datasets
  4. Target-Independent Datasets
  5. Dialogue Summarization Datasets
  6. Argumentative Dialogue Summarization Datasets
  7. Creating OrChiD
  8. Data Collection
  9. ASR-aided Transcription
  10. Automated and Manual Annotation
  11. Topic and Position Statement
  12. Debate Segmentation
  13. Stance and Debater
  14. Reference Summaries
  15. Quality Control
  16. ...and 26 more sections

Figures (3)

  • Figure 1: An excerpt of one debate in OrChiD. One debate entry of our dataset consists of: (1) debate topic, (2) position statements of both sides, (3) utterances labelled with speaker and stance, and (4) stance-specific summaries. Original text is in Chinese (see Appendix \ref{['appendix:full_example']} for a more complete example).
  • Figure 2: Distribution of topic domains in OrChiD. While there are 476 unique topics, one topic could be classified into multiple domains.
  • Figure 3: Comparison of R-1 scores of four methods on both overall summarization and stance-specific summarization tasks. DnS: Divide-and-Summarize; A.C.E.: Accumulative Context Enhanced; I.R.: Iterative Revision.