Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Large-Scale Corpus for Conversation Disentanglement

Jonathan K. Kummerfeld, Sai R. Gouravajhala, Joseph Peper, Vignesh Athreya, Chulaka Gunasekara, Jatin Ganhotra, Siva Sankalp Patel, Lazaros Polymenakos, Walter S. Lasecki

TL;DR

The paper tackles conversation disentanglement in multi-party chat by introducing a large, context-inclusive IRC dataset with manually annotated reply-to graphs and adjudicated disagreements. It formalizes the task, compares datasets, and demonstrates the value of adjudication and contextual information for high-quality evaluation. Through a suite of models and extensive cross-dataset analysis, it shows that prior large-scale heuristics misrepresent real dialogues and that diverse training data improves robustness for downstream tasks like next-utterance prediction. The dataset and findings provide a crucial resource for advancing robust, data-driven disentanglement methods and related dialogue research.

Abstract

Disentangling conversations mixed together in a single stream of messages is a difficult task, made harder by the lack of large manually annotated datasets. We created a new dataset of 77,563 messages manually annotated with reply-structure graphs that both disentangle conversations and define internal conversation structure. Our dataset is 16 times larger than all previously released datasets combined, the first to include adjudication of annotation disagreements, and the first to include context. We use our data to re-examine prior work, in particular, finding that 80% of conversations in a widely used dialogue corpus are either missing messages or contain extra messages. Our manually-annotated data presents an opportunity to develop robust data-driven methods for conversation disentanglement, which will help advance dialogue research.

A Large-Scale Corpus for Conversation Disentanglement

TL;DR

The paper tackles conversation disentanglement in multi-party chat by introducing a large, context-inclusive IRC dataset with manually annotated reply-to graphs and adjudicated disagreements. It formalizes the task, compares datasets, and demonstrates the value of adjudication and contextual information for high-quality evaluation. Through a suite of models and extensive cross-dataset analysis, it shows that prior large-scale heuristics misrepresent real dialogues and that diverse training data improves robustness for downstream tasks like next-utterance prediction. The dataset and findings provide a crucial resource for advancing robust, data-driven disentanglement methods and related dialogue research.

Abstract

Disentangling conversations mixed together in a single stream of messages is a difficult task, made harder by the lack of large manually annotated datasets. We created a new dataset of 77,563 messages manually annotated with reply-structure graphs that both disentangle conversations and define internal conversation structure. Our dataset is 16 times larger than all previously released datasets combined, the first to include adjudication of annotation disagreements, and the first to include context. We use our data to re-examine prior work, in particular, finding that 80% of conversations in a widely used dialogue corpus are either missing messages or contain extra messages. Our manually-annotated data presents an opportunity to develop robust data-driven methods for conversation disentanglement, which will help advance dialogue research.

Paper Structure

This paper contains 13 sections, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Task Definition
  3. Related Work
  4. Data
  5. Dataset Comparison
  6. Methodology
  7. Annotation Quality
  8. Evaluating Disentanglement Quality
  9. Results
  10. Channel Two Results
  11. Evaluating Lowe:2015Lowe:2017:DD
  12. Re-Examining Disentanglement Research
  13. Conclusion

Figures (4)

  • Figure 1: #Ubuntu IRC log sample, earliest message first. Curved lines are our graph annotations of reply structure, which define two conversations shown with blue solid edges and green dashed edges.
  • Figure 2: Examples of annotation ambiguity. Top: The message from MOUD could be a response to either nacc or MonkeyDust. Bottom: The message from Madsy could be part of this conversation or a separate exchange between the same users.
  • Figure 3: An example conversation extracted by the heuristic from Lowe:2015Lowe:2017:DD with the messages it misses and the one it incorrectly includes.
  • Figure 4: Time between consecutive messages in conversations. Jumps are at points when the scale shifts as indicated on the x-axis. The circled upper right point is the sum over all larger values, indicating that messages weeks apart are often in the same conversation.