Deaf and Hard of Hearing Access to Intelligent Personal Assistants: Comparison of Voice-Based Options with an LLM-Powered Touch Interface

TL;DR

This paper tackles the accessibility bottleneck in intelligent personal assistants for Deaf and Hard of Hearing users who speak. It compares three input modalities—natural deaf speech with built-in ASR, Wizard-of-Oz facilitated deaf-accented English speech, and an LLM-powered touch interface—to assess usability and satisfaction in a mixed-methods study. Quantitative results show no significant differences among conditions on SUS, Adjective Scale, and NPS, while qualitative data reveal strong preferences for hands-free and multimodal options, with latency and re-speaking delays posing challenges. The findings highlight the need for native, robust deaf-speech support in IPAs and suggest that context-aware LLM-assisted interfaces can complement speech, provided latency is addressed and sign-language input options are explored for broader accessibility.

Abstract

We investigate intelligent personal assistants (IPAs) accessibility for deaf and hard of hearing (DHH) people who can use their voice in everyday communication. The inability of IPAs to understand diverse accents including deaf speech renders them largely inaccessible to non-signing and speaking DHH individuals. Using an Echo Show, we compare the usability of natural language input via spoken English; with Alexa's automatic speech recognition and a Wizard-of-Oz setting with a trained facilitator re-speaking commands against that of a large language model (LLM)-assisted touch interface in a mixed-methods study. The touch method was navigated through an LLM-powered "task prompter," which integrated the user's history and smart environment to suggest contextually-appropriate commands. Quantitative results showed no significant differences across both spoken English conditions vs LLM-assisted touch. Qualitative results showed variability in opinions on the usability of each method. Ultimately, it will be necessary to have robust deaf-accented speech recognized natively by IPAs.

Figures (10)

• Figure 1: A user selecting LLM-populated options on a touchscreen in our LLM Touch interface.
• Figure 2: System overview. The user interacts with a tablet which uses Web technologies to communicate with an LLM backend to iteratively generate the Alexa command.
• Figure 3: Mean SUS scores for all three conditions.
• Figure 4: Scatterplot of SUS for all three conditions. The height bars indicate the SUS for the LLM-assisted touch condition, while the x and y coordinates of each point indicate the SUS for the Natural Deaf Speech and Facilitated English conditions for each participant, respectively.
• Figure 5: Mean Adjective Scale scores for all three conditions.