Modeling the Quality of Dialogical Explanations

TL;DR

This paper investigates how the quality of dialogical explanations correlates with interactive patterns between explainers and explainees by building a corpus of 399 daily-life explanation dialogues from the Reddit ELI5 forum and annotating each turn for explanation moves, dialogue acts, and topic relations, plus a 5-point explanation-quality score. It contrasts these everyday dialogues with expert dialogues and demonstrates that encoding interaction flows as per-turn tokens enhances the ability of long-input models (Longformer and Hierarchical Attention Transformer) to predict explanation success, with HatFormer achieving the best RMSE of 1.28 and MAE of 1.05 when using all labels. The study also shows robust cross-domain generalization and promising early-prediction capabilities, indicating practical utility for guiding explainers in real time. The authors release the corpus and code publicly and discuss ethical considerations related to data use and annotation practices, highlighting future directions such as alternative architectures and prompting LLMs for improved modeling of dialogical explanations.

Abstract

Explanations are pervasive in our lives. Mostly, they occur in dialogical form where an {\em explainer} discusses a concept or phenomenon of interest with an {\em explainee}. Leaving the explainee with a clear understanding is not straightforward due to the knowledge gap between the two participants. Previous research looked at the interaction of explanation moves, dialogue acts, and topics in successful dialogues with expert explainers. However, daily-life explanations often fail, raising the question of what makes a dialogue successful. In this work, we study explanation dialogues in terms of the interactions between the explainer and explainee and how they correlate with the quality of explanations in terms of a successful understanding on the explainee's side. In particular, we first construct a corpus of 399 dialogues from the Reddit forum {\em Explain Like I am Five} and annotate it for interaction flows and explanation quality. We then analyze the interaction flows, comparing them to those appearing in expert dialogues. Finally, we encode the interaction flows using two language models that can handle long inputs, and we provide empirical evidence for the effectiveness boost gained through the encoding in predicting the success of explanation dialogues.

Figures (4)

• Figure 1: Example explanation dialogue from the ELI5 corpus introduced in this paper, annotated for explanation moves, dialogue acts, and topics
• Figure 2: Our approach is to augment the input of language models with tokens reflecting the interaction flow in terms of either the explanation moves, dialogue acts, topic, or all together. Here, it is shown for the case of explanation moves.
• Figure 3: Explanation moves, dialogue acts, and topics distributions in our corpus and the 5-Levels the expert dialogues corpus of wachsmuth:2022
• Figure 4: The root mean squared error (RSME) of all models for early predictions of explanation quality, that is, when the input to the models is only a defined initial percentage (10%, $\ldots$, 100%) of the full explanation dialogue.