Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SpeechAgent: An End-to-End Mobile Infrastructure for Speech Impairment Assistance

Haowei Lou, Chengkai Huang, Hye-young Paik, Yongquan Hu, Aaron Quigley, Wen Hu, Lina Yao

TL;DR

SpeechAgent introduces a mobile edge–cloud system that jointly reasons about impaired speech and generates clearer, intention-preserving output in real time. By integrating a dedicated Speech Impairment Recognition model, ASR, LLM-based refinement conditioned on impairment type, and a ParaStyleTTS-based synthesis module, the approach enables end-to-end perception–reasoning–generation for everyday communication. The authors provide a multi-source dataset and a benchmarking pipeline, demonstrating strong impairment classification performance (e.g., Transformer-based SIR with ROC-AUC ≈ 0.99) and effective text- and speech-based refinement, while maintaining low latency (RTF ≈ 0.08). The work highlights practical viability for on-device interaction via edge servers, with clear pathways for future improvements in semantic fidelity, interactivity, and multilingual personalization.

Abstract

Speech is essential for human communication, yet millions of people face impairments such as dysarthria, stuttering, and aphasia conditions that often lead to social isolation and reduced participation. Despite recent progress in automatic speech recognition (ASR) and text-to-speech (TTS) technologies, accessible web and mobile infrastructures for users with impaired speech remain limited, hindering the practical adoption of these advances in daily communication. To bridge this gap, we present SpeechAgent, a mobile SpeechAgent designed to facilitate people with speech impairments in everyday communication. The system integrates large language model (LLM)- driven reasoning with advanced speech processing modules, providing adaptive support tailored to diverse impairment types. To ensure real-world practicality, we develop a structured deployment pipeline that enables real-time speech processing on mobile and edge devices, achieving imperceptible latency while maintaining high accuracy and speech quality. Evaluation on real-world impaired speech datasets and edge-device latency profiling confirms that SpeechAgent delivers both effective and user-friendly performance, demonstrating its feasibility for personalized, day-to-day assistive communication.

SpeechAgent: An End-to-End Mobile Infrastructure for Speech Impairment Assistance

TL;DR

SpeechAgent introduces a mobile edge–cloud system that jointly reasons about impaired speech and generates clearer, intention-preserving output in real time. By integrating a dedicated Speech Impairment Recognition model, ASR, LLM-based refinement conditioned on impairment type, and a ParaStyleTTS-based synthesis module, the approach enables end-to-end perception–reasoning–generation for everyday communication. The authors provide a multi-source dataset and a benchmarking pipeline, demonstrating strong impairment classification performance (e.g., Transformer-based SIR with ROC-AUC ≈ 0.99) and effective text- and speech-based refinement, while maintaining low latency (RTF ≈ 0.08). The work highlights practical viability for on-device interaction via edge servers, with clear pathways for future improvements in semantic fidelity, interactivity, and multilingual personalization.

Abstract

Speech is essential for human communication, yet millions of people face impairments such as dysarthria, stuttering, and aphasia conditions that often lead to social isolation and reduced participation. Despite recent progress in automatic speech recognition (ASR) and text-to-speech (TTS) technologies, accessible web and mobile infrastructures for users with impaired speech remain limited, hindering the practical adoption of these advances in daily communication. To bridge this gap, we present SpeechAgent, a mobile SpeechAgent designed to facilitate people with speech impairments in everyday communication. The system integrates large language model (LLM)- driven reasoning with advanced speech processing modules, providing adaptive support tailored to diverse impairment types. To ensure real-world practicality, we develop a structured deployment pipeline that enables real-time speech processing on mobile and edge devices, achieving imperceptible latency while maintaining high accuracy and speech quality. Evaluation on real-world impaired speech datasets and edge-device latency profiling confirms that SpeechAgent delivers both effective and user-friendly performance, demonstrating its feasibility for personalized, day-to-day assistive communication.
Paper Structure (23 sections, 10 equations, 5 figures, 3 tables, 1 algorithm)

This paper contains 23 sections, 10 equations, 5 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. Methodology
  5. Framework Overview
  6. Speech Impairment Recognition
  7. Speech Recognition
  8. Speech Refinement
  9. System Infrastructure
  10. Edge-Device Functions
  11. Server Functions
  12. Benchmarking and Evaluation
  13. Dataset & Reprocessing
  14. Evaluation
  15. Result and Discussion
  16. ...and 8 more sections

Figures (5)

  • Figure 1: Overview of the SpeechAgent system. The edge device records impaired speech and sends it to the cloud server via HTTP, where the Refine Speech module performs recognition and restoration before returning the refined output. The examples illustrate how SpeechAgent converts dysarthric, stuttered, and aphasic speech into clear, fluent sentences.
  • Figure 2: Overview of the system architecture of the proposed SpeechAgent. The system integrates speech impairment recognition, LLM-based speech refinement, and text-to-speech generation. It operates in an edge–server setup that refines impaired speech into clearer, intention-preserving output for real-time communication.
  • Figure 3: Model-wise inference time for refining speech samples with average duration of 11.49s
  • Figure 4: LLM refine time comparison
  • Figure 5: Mobile SpeechAgent UI Demo.