Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Comparing Perceptions of Static and Adaptive Proactive Speech Agents

Justin Edwards, Philip R. Doyle, Holly P. Branigan, Benjamin R. Cowan

TL;DR

This study tests whether a context-adaptive proactive speech agent improves user perceptions compared with a static, non-adaptive agent by examining interruption timing, appropriateness, and partner-model strength using the Partner Modelling Questionnaire (PMQ) across TikTok-like Tetris video stimuli. Using a within-subject design with 80 participants, it manipulates adaptive versus static interruption strategies and measures responses via PMQ and post-trial items; analyses rely on linear mixed-effects models. Contrary to predictions, the adaptive agent did not achieve better timing or appropriateness and elicited weaker partner models, driven mainly by perceived declines in competence and dependability. The findings challenge the assumption that human-inspired adaptivity automatically improves perceived dialogic quality, underscoring the importance of consistency, social appropriateness, and clear exposure when deploying adaptive features in speech agents.

Abstract

A growing literature on speech interruptions describes how people interrupt one another with speech, but these behaviours have not yet been implemented in the design of artificial agents which interrupt. Perceptions of a prototype proactive speech agent which adapts its speech to both urgency and to the difficulty of the ongoing task it interrupts are compared against perceptions of a static proactive agent which does not. The study hypothesises that adaptive proactive speech modelled on human speech interruptions will lead to partner models which consider the proactive agent as a stronger conversational partner than a static agent, and that interruptions initiated by an adaptive agent will be judged as better timed and more appropriately asked. These hypotheses are all rejected however, as quantitative analysis reveals that participants view the adaptive agent as a poorer dialogue partner than the static agent and as less appropriate in the style it interrupts. Qualitative analysis sheds light on the source of this surprising finding, as participants see the adaptive agent as less socially appropriate and as less consistent in its interactions than the static agent.

Comparing Perceptions of Static and Adaptive Proactive Speech Agents

TL;DR

This study tests whether a context-adaptive proactive speech agent improves user perceptions compared with a static, non-adaptive agent by examining interruption timing, appropriateness, and partner-model strength using the Partner Modelling Questionnaire (PMQ) across TikTok-like Tetris video stimuli. Using a within-subject design with 80 participants, it manipulates adaptive versus static interruption strategies and measures responses via PMQ and post-trial items; analyses rely on linear mixed-effects models. Contrary to predictions, the adaptive agent did not achieve better timing or appropriateness and elicited weaker partner models, driven mainly by perceived declines in competence and dependability. The findings challenge the assumption that human-inspired adaptivity automatically improves perceived dialogic quality, underscoring the importance of consistency, social appropriateness, and clear exposure when deploying adaptive features in speech agents.

Abstract

A growing literature on speech interruptions describes how people interrupt one another with speech, but these behaviours have not yet been implemented in the design of artificial agents which interrupt. Perceptions of a prototype proactive speech agent which adapts its speech to both urgency and to the difficulty of the ongoing task it interrupts are compared against perceptions of a static proactive agent which does not. The study hypothesises that adaptive proactive speech modelled on human speech interruptions will lead to partner models which consider the proactive agent as a stronger conversational partner than a static agent, and that interruptions initiated by an adaptive agent will be judged as better timed and more appropriately asked. These hypotheses are all rejected however, as quantitative analysis reveals that participants view the adaptive agent as a poorer dialogue partner than the static agent and as less appropriate in the style it interrupts. Qualitative analysis sheds light on the source of this surprising finding, as participants see the adaptive agent as less socially appropriate and as less consistent in its interactions than the static agent.
Paper Structure (27 sections, 4 figures, 6 tables)

This paper contains 27 sections, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Designing proactive agents
  4. Partner modelling of machine dialogue partners
  5. Aims and hypotheses
  6. Experimental Method
  7. Participants
  8. Materials
  9. Experimental conditions- Adaptive vs Static Proactive Agent
  10. Adaptive agent
  11. Static agent
  12. Measures
  13. Partner Model Questionnaire
  14. Single Item Questionnaires
  15. Demographic Questionnaire
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Example screenshots from the experiment which participants saw as part of pre-test instructions. On the left, there is no red dot, so the agent has not yet been cued to interrupt. On the right, the red dot has appeared, signalling that the agent has been cued to interrupt.
  • Figure 2: Predicted values of appropriateness questionnaire ratings by condition
  • Figure 3: Predicted values of Partner Model Questionnaire total scores by condition
  • Figure 4: Predicted values of Partner Model Questionnaire partner competence and dependability subscale scores by condition