Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Words: Measuring User Experience through Speech Analysis in Voice User Interfaces

Yong Ma, Xuesong Zhang, Xuedong Zhang, Natalia Bartłomiejczyk, Seungwoo Je, Adrian Holzer, Morten Fjeld, Andreas Butz

Abstract

Voice assistants (VAs) are typically evaluated through task performance metrics and self-report questionnaires, but people's voices themselves carry rich paralinguistic cues that reveal affect, effort, and interaction breakdowns. We present a within-subjects study (N=49) that systematically compared three VA personas across three usage scenarios to investigate whether speech-derived audio features can serve as a proxy for user experience (UX). Participants' speech was analyzed for temporal, spectral, and linguistic markers, alongside standardized UX measures, brief mood and stress ratings, and a post-study questionnaire. We found correlations between specific speech features and self-reported satisfaction and experience. Furthermore, a machine learning model trained on speech features achieved promising accuracy in classifying UX levels, indicating that this might be a reasonable alternative to self-report instruments. Our findings establish speech as a viable, real-time signal for implicitly measuring UX and point toward adaptive VUIs that respond dynamically to emotional and usability-related vocal cues.

Beyond Words: Measuring User Experience through Speech Analysis in Voice User Interfaces

Abstract

Voice assistants (VAs) are typically evaluated through task performance metrics and self-report questionnaires, but people's voices themselves carry rich paralinguistic cues that reveal affect, effort, and interaction breakdowns. We present a within-subjects study (N=49) that systematically compared three VA personas across three usage scenarios to investigate whether speech-derived audio features can serve as a proxy for user experience (UX). Participants' speech was analyzed for temporal, spectral, and linguistic markers, alongside standardized UX measures, brief mood and stress ratings, and a post-study questionnaire. We found correlations between specific speech features and self-reported satisfaction and experience. Furthermore, a machine learning model trained on speech features achieved promising accuracy in classifying UX levels, indicating that this might be a reasonable alternative to self-report instruments. Our findings establish speech as a viable, real-time signal for implicitly measuring UX and point toward adaptive VUIs that respond dynamically to emotional and usability-related vocal cues.
Paper Structure (68 sections, 14 figures, 14 tables)

This paper contains 68 sections, 14 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evaluating Voice User Interfaces
  4. Speech as a Window into User State
  5. Speech-based UX Inference in HCI
  6. System Behaviors as Design Levers
  7. Methods for Extracting Speech Features
  8. Research Gaps
  9. Study Design
  10. Overview of Experimental Procedure
  11. Design of Voice Assistant Personas
  12. LLM and Response-Style Manipulation
  13. Response Latency
  14. Voice Quality
  15. Design of Scenarios
  16. ...and 53 more sections

Figures (14)

  • Figure 1: The interface for the post-study interview, showing one of the four questions administered via the dedicated voice assistant (VA). This VA asked participants to reflect on and compare their experiences with the three experimental personas.
  • Figure 2: The three interaction scenarios as they were presented to participants, each designed to elicit different conversational patterns and user expectations: (a) a functional Trip Planning task, (b) a creative Collaborative Storytelling task, and (c) a playful Fortune Teller task.
  • Figure 3: Overview of participants gender, age and talkativeness.
  • Figure 4: Mean mood ratings from all study rounds by VA (left) and Scenario (right).
  • Figure 5: Mean change from baseline pre-study in Mood (left) and Stress (right) for each VA persona.
  • ...and 9 more figures