LTS-VoiceAgent: A Listen-Think-Speak Framework for Efficient Streaming Voice Interaction via Semantic Triggering and Incremental Reasoning

TL;DR

The paper tackles the challenge of real-time voice agents balancing instant responsiveness with deep reasoning. It introduces LTS-VoiceAgent, a Listen-Think-Speak framework that uses a Dynamic Semantic Trigger to activate reasoning only at semantically complete prefixes and a Dual-Role Stream Orchestrator to run background Thinker state maintenance alongside foreground Speaker speculation. It formalizes a Delta-State CoT workflow and a Pause-and-Repair benchmark to stress test disfluencies, demonstrating through experiments on VERA, Spoken-MQA, BigBenchAudio, and the proposed dataset that the approach outperforms serial cascades and prior streaming strategies in accuracy, latency, and efficiency. The work advances practical interactive AI by enabling "thinking while listening" with robust streaming reasoning, reduced wasteful computation, and resilient performance under fluent and disfluent speech. $P_{trigger}$ surpassing $\tau$ governs when to think, while stateful orchestration preserves reasoning quality across dynamic inputs.

Abstract

Real-time voice agents face a dilemma: end-to-end models often lack deep reasoning, while cascaded pipelines incur high latency by executing ASR, LLM reasoning, and TTS strictly in sequence, unlike human conversation where listeners often start thinking before the speaker finishes. Since cascaded architectures remain the dominant choice for complex tasks, existing cascaded streaming strategies attempt to reduce this latency via mechanical segmentation (e.g., fixed chunks, VAD-based splitting) or speculative generation, but they frequently either break semantic units or waste computation on predictions that must be rolled back. To address these challenges, we propose LTS-VoiceAgent, a Listen-Think-Speak framework that explicitly separates when to think from how to reason incrementally. It features a Dynamic Semantic Trigger to detect meaningful prefixes, and a Dual-Role Stream Orchestrator that coordinates a background Thinker (for state maintenance) and a foreground Speaker (for speculative solving). This parallel design enables "thinking while speaking" without blocking responses. We also introduce a Pause-and-Repair benchmark containing natural disfluencies to stress-test streaming robustness. Experiments across VERA, Spoken-MQA, BigBenchAudio, and our benchmark show that LTS-VoiceAgent achieves a stronger accuracy-latency-efficiency trade-off than serial cascaded baselines and existing streaming strategies.

Figures (7)

• Figure 1: Traditional Serial Processing vs. LTS-VoiceAgent Framework for Voice Interaction.
• Figure 2: Examples of different reasoning methods.
• Figure 3: Overview of LTS-VoiceAgent.
• Figure 4: Performance (Accuracy) vs. Latency trade-off analysis.
• Figure 5: Ablation study on the LTS-VoiceAgent.