Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Partner Modelling Questionnaire: A validated self-report measure of perceptions toward machines as dialogue partners

Philip R. Doyle, Iona Gessinger, Justin Edwards, Leigh Clark, Odile Dumbleton, Diego Garaialde, Daniel Rough, Anna Bleakley, Holly P. Branigan, Benjamin R. Cowan

TL;DR

This paper introduces the Partner Modelling Questionnaire (PMQ), an 18-item self-report scale that measures three dimensions of users’ perceptions of machines as dialogue partners: communicative competence/dependability, human-likeness in communication, and communicative flexibility. Through three phases—item generation, construct validity via CFA, and convergent/divergent validity plus test-retest reliability—the PMQ demonstrates strong internal consistency, robust factor structure, and good stability over 4–12 weeks. The 18-item version outperforms the original 23-item model in CFA, and the PMQ shows convergent validity with usability and anthropomorphism scales while maintaining discriminant validity. The PMQ offers researchers a direct, scalable tool to quantify partner models, enabling more precise investigations into how speech interface design influences user perceptions and subsequent interaction behaviors. Overall, the PMQ facilitates longitudinal studies and cross-domain comparisons, advancing theory and practice in speech interface design and human–machine dialogue.

Abstract

Recent work has looked to understand user perceptions of speech agent capabilities as dialogue partners (termed partner models), and how this affects user interaction. Yet, currently partner model effects are inferred from language production as no metrics are available to quantify these subjective perceptions more directly. Through three studies, we develop and validate the Partner Modelling Questionnaire (PMQ): an 18-item self-report semantic differential scale designed to reliably measure people's partner models of non-embodied speech interfaces. Through principal component analysis and confirmatory factor analysis, we show that the PMQ scale consists of three factors: communicative competence and dependability, human-likeness in communication, and communicative flexibility. Our studies show that the measure consistently demonstrates good internal reliability, strong test-retest reliability over 12 and 4-week intervals, and predictable convergent/divergent validity. Based on our findings we discuss the multidimensional nature of partner models, whilst identifying key future research avenues that the development of the PMQ facilitates. Notably, this includes the need to identify the activation, sensitivity, and dynamism of partner models in speech interface interaction.

The Partner Modelling Questionnaire: A validated self-report measure of perceptions toward machines as dialogue partners

TL;DR

This paper introduces the Partner Modelling Questionnaire (PMQ), an 18-item self-report scale that measures three dimensions of users’ perceptions of machines as dialogue partners: communicative competence/dependability, human-likeness in communication, and communicative flexibility. Through three phases—item generation, construct validity via CFA, and convergent/divergent validity plus test-retest reliability—the PMQ demonstrates strong internal consistency, robust factor structure, and good stability over 4–12 weeks. The 18-item version outperforms the original 23-item model in CFA, and the PMQ shows convergent validity with usability and anthropomorphism scales while maintaining discriminant validity. The PMQ offers researchers a direct, scalable tool to quantify partner models, enabling more precise investigations into how speech interface design influences user perceptions and subsequent interaction behaviors. Overall, the PMQ facilitates longitudinal studies and cross-domain comparisons, advancing theory and practice in speech interface design and human–machine dialogue.

Abstract

Recent work has looked to understand user perceptions of speech agent capabilities as dialogue partners (termed partner models), and how this affects user interaction. Yet, currently partner model effects are inferred from language production as no metrics are available to quantify these subjective perceptions more directly. Through three studies, we develop and validate the Partner Modelling Questionnaire (PMQ): an 18-item self-report semantic differential scale designed to reliably measure people's partner models of non-embodied speech interfaces. Through principal component analysis and confirmatory factor analysis, we show that the PMQ scale consists of three factors: communicative competence and dependability, human-likeness in communication, and communicative flexibility. Our studies show that the measure consistently demonstrates good internal reliability, strong test-retest reliability over 12 and 4-week intervals, and predictable convergent/divergent validity. Based on our findings we discuss the multidimensional nature of partner models, whilst identifying key future research avenues that the development of the PMQ facilitates. Notably, this includes the need to identify the activation, sensitivity, and dynamism of partner models in speech interface interaction.
Paper Structure (53 sections, 3 figures, 8 tables)

This paper contains 53 sections, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. The concept of partner models
  4. Measuring partner models in hmd
  5. Subjective measures in hmd research
  6. Research aims
  7. Phase 1: Item generation and scale construction
  8. Item pool generation and reduction
  9. Stage 1 -- Repertory Grid Study
  10. Stage 2 -- Systematic review of subjective measures
  11. Stage 3 -- Item pool screening
  12. Initial factor structure
  13. Phase 2: Construct Validity
  14. Study 1 Participants
  15. Study 1 Procedure
  16. ...and 38 more sections

Figures (3)

  • Figure 1: Path diagram of CFA output for 23-item 3-factor model, with factor loadings and between factor correlation coefficients
  • Figure 2: Path diagram of CFA output for 18-item 3-factor model, with factor loadings and between-factor correlation coefficients
  • Figure 3: Path diagram of CFA output for 23-item 3-factor model, with factor loadings and between-factor correlation coefficients