KoCoSa: Korean Context-aware Sarcasm Detection Dataset

TL;DR

A new dataset for the Korean dialogue sarcasm detection task, KoCoSa (Korean Context-aware Sarcasm Detection Dataset), which consists of 12.8K daily Korean dialogues and the labels for this task on the last response is introduced.

Abstract

Sarcasm is a way of verbal irony where someone says the opposite of what they mean, often to ridicule a person, situation, or idea. It is often difficult to detect sarcasm in the dialogue since detecting sarcasm should reflect the context (i.e., dialogue history). In this paper, we introduce a new dataset for the Korean dialogue sarcasm detection task, KoCoSa (Korean Context-aware Sarcasm Detection Dataset), which consists of 12.8K daily Korean dialogues and the labels for this task on the last response. To build the dataset, we propose an efficient sarcasm detection dataset generation pipeline: 1) generating new sarcastic dialogues from source dialogues with large language models, 2) automatic and manual filtering of abnormal and toxic dialogues, and 3) human annotation for the sarcasm detection task. We also provide a simple but effective baseline for the Korean sarcasm detection task trained on our dataset. Experimental results on the dataset show that our baseline system outperforms strong baselines like large language models, such as GPT-3.5, in the Korean sarcasm detection task. We show that the sarcasm detection task relies deeply on the existence of sufficient context. We will release the dataset at https://github.com/Yu-billie/KoCoSa_sarcasm_detection.

Figures (4)

• Figure 1: Examples on Korean sarcasm detection results for a target utterance, (a) without the context and (b) with context, respectively.
• Figure 2: The overall pipeline of KoCoSa dataset construction. In the Generated Dialogue example, light blue letters represent the honorifics ending of a word in Korean. This figure is best viewed in color.
• Figure 3: Topic diversity of Online Text Message Corpus and Messenger Corpus. Topics that account for less than 5% are grouped as Misc.
• Figure 4: The task description and data labeling web page for the annotation process. For better understanding, we provide the translation of the Task Description in the figure. Task Description: We are conducting research on constructing a Korean Sarcasm Dataset. In this survey, we would like you to verify the validity of the provided sarcastic sentences by checking the three items below. The three items refer to the annotation guidelines in section \ref{['subsubsec:annotation_guideline']}.