Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AsyncVoice Agent: Real-Time Explanation for LLM Planning and Reasoning

Yueqian Lin, Zhengmian Hu, Jayakumar Subramanian, Qinsi Wang, Nikos Vlassis, Hai "Helen" Li, Yiran Chen

TL;DR

The paper addresses the challenge of making LLM reasoning transparent and interactable in real-time within spoken interfaces, where traditional monolithic Chain-of-Thought outputs hinder collaboration. It introduces AsyncVoice Agent, an asynchronous architecture that decouples a streaming backend (MCP servers) from a conversational voice frontend to narrate live reasoning while allowing user interruptions. Key contributions include the MCP-based modular reasoning framework, a bidirectional WebSocket protocol, a four-thread speech pipeline with Azure TTS crossfading, and a DistilBERT-based turn detector for natural interruption handling, all validated against automated benchmarks showing sub-second responsiveness and substantial latency reductions. The work demonstrates a practical path toward interactive, explainable, and trustworthy human-AI systems for high-stakes tasks by transforming CoT from static text into a steerable, real-time dialogue.

Abstract

Effective human-AI collaboration on complex reasoning tasks requires that users understand and interact with the model's process, not just receive an output. However, the monolithic text from methods like Chain-of-Thought (CoT) prevents this, as current interfaces lack real-time verbalization and robust user barge-in. We present AsyncVoice Agent, a system whose asynchronous architecture decouples a streaming LLM backend from a conversational voice frontend. This design allows narration and inference to run in parallel, empowering users to interrupt, query, and steer the model's reasoning process at any time. Objective benchmarks show this approach reduces interaction latency by more than 600x compared to monolithic baselines while ensuring high fidelity and competitive task accuracy. By enabling a two-way dialogue with a model's thought process, AsyncVoice Agent offers a new paradigm for building more effective, steerable, and trustworthy human-AI systems for high-stakes tasks.

AsyncVoice Agent: Real-Time Explanation for LLM Planning and Reasoning

TL;DR

The paper addresses the challenge of making LLM reasoning transparent and interactable in real-time within spoken interfaces, where traditional monolithic Chain-of-Thought outputs hinder collaboration. It introduces AsyncVoice Agent, an asynchronous architecture that decouples a streaming backend (MCP servers) from a conversational voice frontend to narrate live reasoning while allowing user interruptions. Key contributions include the MCP-based modular reasoning framework, a bidirectional WebSocket protocol, a four-thread speech pipeline with Azure TTS crossfading, and a DistilBERT-based turn detector for natural interruption handling, all validated against automated benchmarks showing sub-second responsiveness and substantial latency reductions. The work demonstrates a practical path toward interactive, explainable, and trustworthy human-AI systems for high-stakes tasks by transforming CoT from static text into a steerable, real-time dialogue.

Abstract

Effective human-AI collaboration on complex reasoning tasks requires that users understand and interact with the model's process, not just receive an output. However, the monolithic text from methods like Chain-of-Thought (CoT) prevents this, as current interfaces lack real-time verbalization and robust user barge-in. We present AsyncVoice Agent, a system whose asynchronous architecture decouples a streaming LLM backend from a conversational voice frontend. This design allows narration and inference to run in parallel, empowering users to interrupt, query, and steer the model's reasoning process at any time. Objective benchmarks show this approach reduces interaction latency by more than 600x compared to monolithic baselines while ensuring high fidelity and competitive task accuracy. By enabling a two-way dialogue with a model's thought process, AsyncVoice Agent offers a new paradigm for building more effective, steerable, and trustworthy human-AI systems for high-stakes tasks.
Paper Structure (13 sections, 2 figures, 1 table)

This paper contains 13 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. AsyncVoice Agent Architecture
  3. WebSocket Communication Infrastructure
  4. MCP-Based Modular Reasoning Architecture
  5. Multi-Threaded Speech Processing Pipeline
  6. Turn Detection and Interruption Handling
  7. Frontend Real-Time Audio Processing
  8. Evaluation
  9. Experimental Setup
  10. Evaluation Scenarios and Metrics
  11. Results
  12. Discussion and Limitations
  13. Conclusion

Figures (2)

  • Figure 1: The high-level architecture of the AsyncVoice Agent. The system features two primary components: the Backend MCP Servers that generate a Continuous Stream of reasoning steps, and the AsyncVoice Agent pipeline, which verbalizes these steps and manages asynchronous user interruptions via a Stop Signal.
  • Figure 2: The front-end web interface for the AsyncVoice Agent. The user can enter a query to begin a task. The interface displays the live Planning Process and indicates that the microphone is always active for interruptions.