Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

It Couldn't Help But Overhear: On the Limits of Modelling Meta-Communicative Grounding Acts with Supervised Learning

Brielen Madureira, David Schlangen

TL;DR

The paper investigates the limits of modelling meta-communicative grounding acts through data-driven supervised learning using the overhearing paradigm. It analyzes how data collection, annotation, modelling, and evaluation practices embed overhearing biases and discusses the fundamental variability of human grounding acts. A pilot study on Clarification Requests (CRs) in CoDraw reveals substantial variability and low inter-annotator agreement, suggesting that current data-driven approaches may fail to capture the true dynamics of grounding. The authors advocate for interactive or reinforcement learning frameworks, hybrid methods, and explicit acknowledgement of the overhearing assumption to better model joint grounding in dialogue and to avoid cognitive misrepresentations of human understanding.

Abstract

Active participation in a conversation is key to building common ground, since understanding is jointly tailored by producers and recipients. Overhearers are deprived of the privilege of performing grounding acts and can only conjecture about intended meanings. Still, data generation and annotation, modelling, training and evaluation of NLP dialogue models place reliance on the overhearing paradigm. How much of the underlying grounding processes are thereby forfeited? As we show, there is evidence pointing to the impossibility of properly modelling human meta-communicative acts with data-driven learning models. In this paper, we discuss this issue and provide a preliminary analysis on the variability of human decisions for requesting clarification. Most importantly, we wish to bring this topic back to the community's table, encouraging discussion on the consequences of having models designed to only "listen in".

It Couldn't Help But Overhear: On the Limits of Modelling Meta-Communicative Grounding Acts with Supervised Learning

TL;DR

The paper investigates the limits of modelling meta-communicative grounding acts through data-driven supervised learning using the overhearing paradigm. It analyzes how data collection, annotation, modelling, and evaluation practices embed overhearing biases and discusses the fundamental variability of human grounding acts. A pilot study on Clarification Requests (CRs) in CoDraw reveals substantial variability and low inter-annotator agreement, suggesting that current data-driven approaches may fail to capture the true dynamics of grounding. The authors advocate for interactive or reinforcement learning frameworks, hybrid methods, and explicit acknowledgement of the overhearing assumption to better model joint grounding in dialogue and to avoid cognitive misrepresentations of human understanding.

Abstract

Active participation in a conversation is key to building common ground, since understanding is jointly tailored by producers and recipients. Overhearers are deprived of the privilege of performing grounding acts and can only conjecture about intended meanings. Still, data generation and annotation, modelling, training and evaluation of NLP dialogue models place reliance on the overhearing paradigm. How much of the underlying grounding processes are thereby forfeited? As we show, there is evidence pointing to the impossibility of properly modelling human meta-communicative acts with data-driven learning models. In this paper, we discuss this issue and provide a preliminary analysis on the variability of human decisions for requesting clarification. Most importantly, we wish to bring this topic back to the community's table, encouraging discussion on the consequences of having models designed to only "listen in".
Paper Structure (14 sections, 2 figures)

This paper contains 14 sections, 2 figures.

Table of Contents

  1. Is Grounding "Supervisable"?
  2. Overhearers in a Conversation
  3. Are NLP Models Only Listening In?
  4. Data Collection
  5. Annotation and Analysis
  6. Modelling
  7. Evaluation
  8. Variability in Human Grounding Acts
  9. A Brief Analysis of Regularity in CRs
  10. Discussion
  11. To conclude
  12. Appendix
  13. Annotation Task
  14. Additional Details

Figures (2)

  • Figure 1: Variability of clarification requests produced by three overhearers in comparison to the original one, in an instance of the instruction-following CoDraw dialogue game (https://creativecommons.org/licenses/by-nc/4.0/), with cliparts from zitnick2013bringing.
  • Figure 2: Variability in the decision of when to request clarification, comparing the decision of the original player with those of three overhearer annotators over 90 instances (horizontal axis) of the CoDraw game. Each cell is a data point and columns correspond to decisions on the same instance.