Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SemEval 2024 -- Task 10: Emotion Discovery and Reasoning its Flip in Conversation (EDiReF)

Shivani Kumar, Md Shad Akhtar, Erik Cambria, Tanmoy Chakraborty

TL;DR

SemEval-2024 Task 10 (EDiReF) delivers a comprehensive study of Emotion Recognition in Conversation (ERC) and Emotion Flip Reasoning (EFR) in English and Hindi-English code-mixed dialogue. It introduces three subtasks across code-mixed and monolingual data, accompanied by two new bilingual corpora (MELD-FR and E-MaSaC/EFR-MaSaC) and explicit trigger-labeling guidelines. Across 84 participants and 24 system-descriptions, top results reached F1 scores of approximately 0.78–0.79 for ERC and EFR tasks, with LLMs leading in code-mixed settings and classical ML methods proving competitive in trigger identification. The work highlights practical implications for dialogue systems, such as translating code-mixed data, leveraging context for emotion flips, and developing interpretable triggers, while providing publicly available datasets to spur future research in multilingual ERC and EFR.

Abstract

We present SemEval-2024 Task 10, a shared task centred on identifying emotions and finding the rationale behind their flips within monolingual English and Hindi-English code-mixed dialogues. This task comprises three distinct subtasks - emotion recognition in conversation for code-mixed dialogues, emotion flip reasoning for code-mixed dialogues, and emotion flip reasoning for English dialogues. Participating systems were tasked to automatically execute one or more of these subtasks. The datasets for these tasks comprise manually annotated conversations focusing on emotions and triggers for emotion shifts (The task data is available at https://github.com/LCS2-IIITD/EDiReF-SemEval2024.git). A total of 84 participants engaged in this task, with the most adept systems attaining F1-scores of 0.70, 0.79, and 0.76 for the respective subtasks. This paper summarises the results and findings from 24 teams alongside their system descriptions.

SemEval 2024 -- Task 10: Emotion Discovery and Reasoning its Flip in Conversation (EDiReF)

TL;DR

SemEval-2024 Task 10 (EDiReF) delivers a comprehensive study of Emotion Recognition in Conversation (ERC) and Emotion Flip Reasoning (EFR) in English and Hindi-English code-mixed dialogue. It introduces three subtasks across code-mixed and monolingual data, accompanied by two new bilingual corpora (MELD-FR and E-MaSaC/EFR-MaSaC) and explicit trigger-labeling guidelines. Across 84 participants and 24 system-descriptions, top results reached F1 scores of approximately 0.78–0.79 for ERC and EFR tasks, with LLMs leading in code-mixed settings and classical ML methods proving competitive in trigger identification. The work highlights practical implications for dialogue systems, such as translating code-mixed data, leveraging context for emotion flips, and developing interpretable triggers, while providing publicly available datasets to spur future research in multilingual ERC and EFR.

Abstract

We present SemEval-2024 Task 10, a shared task centred on identifying emotions and finding the rationale behind their flips within monolingual English and Hindi-English code-mixed dialogues. This task comprises three distinct subtasks - emotion recognition in conversation for code-mixed dialogues, emotion flip reasoning for code-mixed dialogues, and emotion flip reasoning for English dialogues. Participating systems were tasked to automatically execute one or more of these subtasks. The datasets for these tasks comprise manually annotated conversations focusing on emotions and triggers for emotion shifts (The task data is available at https://github.com/LCS2-IIITD/EDiReF-SemEval2024.git). A total of 84 participants engaged in this task, with the most adept systems attaining F1-scores of 0.70, 0.79, and 0.76 for the respective subtasks. This paper summarises the results and findings from 24 teams alongside their system descriptions.
Paper Structure (22 sections, 3 figures, 7 tables)

This paper contains 22 sections, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Emotion recognition.
  4. Emotion and code-mixing.
  5. Beyond emotion recognition.
  6. Data
  7. English Conversations:
  8. Hindi-English Code-mixed Conversations:
  9. Task and Background
  10. Shared Task Settings
  11. Pilot Study
  12. Participants
  13. Using LLMs
  14. Classical machine learning and deep learning methods
  15. Rule-based and statistical methods
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Examples of emotion-flip reasoning.
  • Figure 2: Distribution of triggers for the last four utterances from the trigger utterance $i$.
  • Figure 3: Emotion distribution in E-MaSaC. The colors depict the distribution of emotions capturing positive, negative, mixed, and no feelings (Abbreviations: Ang: Anger, Cnt: Contempt, Dis: Disgust, Fea: Fear, Ntr: Neutral, Sad: Sadness, Sur: Surprise).