Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WHoW: A Cross-domain Approach for Analysing Conversation Moderation

Ming-Bin Chen, Lea Frermann, Jey Han Lau

TL;DR

WHoW, an evaluation framework for analyzing the facilitation strategies of moderators across different domains/scenarios by examining their motives, dialogue acts and target speaker, enhances the understanding of moderation behaviour through automatic large-scale analysis and facilitates the development of moderator agents.

Abstract

We propose WHoW, an evaluation framework for analyzing the facilitation strategies of moderators across different domains/scenarios by examining their motives (Why), dialogue acts (How) and target speaker (Who). Using this framework, we annotated 5,657 moderation sentences with human judges and 15,494 sentences with GPT-4o from two domains: TV debates and radio panel discussions. Comparative analysis demonstrates the framework's cross-domain generalisability and reveals distinct moderation strategies: debate moderators emphasise coordination and facilitate interaction through questions and instructions, while panel discussion moderators prioritize information provision and actively participate in discussions. Our analytical framework works for different moderation scenarios, enhances our understanding of moderation behaviour through automatic large-scale analysis, and facilitates the development of moderator agents.

WHoW: A Cross-domain Approach for Analysing Conversation Moderation

TL;DR

WHoW, an evaluation framework for analyzing the facilitation strategies of moderators across different domains/scenarios by examining their motives, dialogue acts and target speaker, enhances the understanding of moderation behaviour through automatic large-scale analysis and facilitates the development of moderator agents.

Abstract

We propose WHoW, an evaluation framework for analyzing the facilitation strategies of moderators across different domains/scenarios by examining their motives (Why), dialogue acts (How) and target speaker (Who). Using this framework, we annotated 5,657 moderation sentences with human judges and 15,494 sentences with GPT-4o from two domains: TV debates and radio panel discussions. Comparative analysis demonstrates the framework's cross-domain generalisability and reveals distinct moderation strategies: debate moderators emphasise coordination and facilitate interaction through questions and instructions, while panel discussion moderators prioritize information provision and actively participate in discussions. Our analytical framework works for different moderation scenarios, enhances our understanding of moderation behaviour through automatic large-scale analysis, and facilitates the development of moderator agents.

Paper Structure

This paper contains 24 sections, 2 equations, 8 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The WHoW Conversational Moderation Analytic Framework
  4. Motives: Why does the moderator intervene?
  5. Dialogue Acts: How does the moderator intervene?
  6. Target Speaker: Who does the moderator address?
  7. Dataset and Human Annotation
  8. Datasets
  9. Human annotation process
  10. Automatic Annotation
  11. Analysis
  12. Motives and Dialogue Acts
  13. Balancing Speaker Participation
  14. Moderators' selection of target speakers
  15. Conclusion
  16. ...and 9 more sections

Figures (8)

  • Figure 1: Example of a moderated conversation and annotation using the WHoW framework. Blue, green, and red colors represent the supporting team, moderator, and opposing team in one of the debate subset conversation, respectively. The peach-colored boxes contain the annotations for the corresponding moderator sentences.
  • Figure 2: Probabilities of participants' rotation statuses following different moderation dialogue acts.
  • Figure 3: The decision tree used by annotators to resolve ambiguous sentences that may involve multiple dialogue acts.
  • Figure 4: Prompt structure and development cycle
  • Figure 5: The normalized co-occurrence matrix of dialogue act human votes from the debate subset.
  • ...and 3 more figures