Computational Analysis of Conversation Dynamics through Participant Responsivity

TL;DR

This work introduces responsivity as a core measure of dialogue quality, proposing semantic-similarity and LLM-based methods to detect whether a speaker's turn responds to preceding turns. It finds that LLM-based annotations align more closely with human judgments than semantic similarity, and it uses these mappings to derive a suite of conversation-level metrics. By clustering conversations into five distinct types, the study demonstrates that responsivity patterns track with facilitation style and conversational purpose, offering a framework for assessing and designing constructive discourse in civic and multi-party settings. The findings have implications for both offline facilitation and online moderation, suggesting practical pathways to foster more pro-social, engaging conversations while noting limitations and avenues for future expansion.

Abstract

Growing literature explores toxicity and polarization in discourse, with comparatively less work on characterizing what makes dialogue prosocial and constructive. We explore conversational discourse and investigate a method for characterizing its quality built upon the notion of ``responsivity'' -- whether one person's conversational turn is responding to a preceding turn. We develop and evaluate methods for quantifying responsivity -- first through semantic similarity of speaker turns, and second by leveraging state-of-the-art large language models (LLMs) to identify the relation between two speaker turns. We evaluate both methods against a ground truth set of human-annotated conversations. Furthermore, selecting the better performing LLM-based approach, we characterize the nature of the response -- whether it responded to that preceding turn in a substantive way or not. We view these responsivity links as a fundamental aspect of dialogue but note that conversations can exhibit significantly different responsivity structures. Accordingly, we then develop conversation-level derived metrics to address various aspects of conversational discourse. We use these derived metrics to explore other conversations and show that they support meaningful characterizations and differentiations across a diverse collection of conversations.

Figures (9)

• Figure 1: Conversation Map showing the flow of turns, their sequence, and the responsivity links.
• Figure 2: Inter-annotator agreement matrix between annotation methods. Note: fs refers to few shot and cons refers to consolidated.
• Figure 3: Human inter-annotator agreement matrix for conversation 1113.
• Figure 4: Cluster Map showing the 5 clusters.
• Figure 5: Heatmap showing the average feature values by cluster.