Speech-Hands: A Self-Reflection Voice Agentic Approach to Speech Recognition and Audio Reasoning with Omni Perception

TL;DR

Speech-Hands tackles the fragility of naive omni-modal fine-tuning by introducing a self-reflective agent that explicitly decides when to trust internal ASR, defer to external hypotheses, or rewrite outputs using a learnable action-token policy. The method unifies ASR and audio reasoning tasks and demonstrates strong improvements, achieving a $12.1\%$ relative WER reduction across seven OpenASR benchmarks and $77.37\%$ average accuracy on multi-domain audio QA. Key contributions include a principled token-based arbitration mechanism, datasets and training protocols enabling cross-task evaluation, and analyses showing high precision for internal/external decisions and robust, albeit scarcer, rewrite triggering. The work offers a practical path toward more reliable and interpretable audio intelligence by fusing perception and decision-making, with potential extensions to multiple external tools and active perception. The reported results establish Speech-Hands as a competitive framework for robust audio understanding with transparent agentic behavior.

Abstract

We introduce a voice-agentic framework that learns one critical omni-understanding skill: knowing when to trust itself versus when to consult external audio perception. Our work is motivated by a crucial yet counterintuitive finding: naively fine-tuning an omni-model on both speech recognition and external sound understanding tasks often degrades performance, as the model can be easily misled by noisy hypotheses. To address this, our framework, Speech-Hands, recasts the problem as an explicit self-reflection decision. This learnable reflection primitive proves effective in preventing the model from being derailed by flawed external candidates. We show that this agentic action mechanism generalizes naturally from speech recognition to complex, multiple-choice audio reasoning. Across the OpenASR leaderboard, Speech-Hands consistently outperforms strong baselines by 12.1% WER on seven benchmarks. The model also achieves 77.37% accuracy and high F1 on audio QA decisions, showing robust generalization and reliability across diverse audio question answering datasets. By unifying perception and decision-making, our work offers a practical path toward more reliable and resilient audio intelligence.

Figures (5)

• Figure 1: Speech-Hands acts as a dynamic orchestrator that predicts a special action token to govern its cognitive strategy for ASR and multi-domain audio reasoning.
• Figure 2: Overview of our proposed Self-Reflection Multimodal GER framework. A special token is generated at the beginning to decide whether to use audio perception (i.e., transcription hypotheses or caption) or not.
• Figure 3: Preliminary results on the cascaded agentic Qwen-omni baseline for generative error correction (GER) with supervised fine-tuning show that both text-only and text-audio GER degrade ASR performance, where the best ASR and LLM combination achieves a low agentic output oracle of 5% WER.
• Figure 4: Confusion matrices of Speech-Hands' agentic action execution for audio QA and reasoning three subsets on (a) bio-acoustic QA, (b) temporal and sound event QA, and (c) complex audio information QA.
• Figure 5: Text-base GER uses only ASR hypotheses. This setup fails to correct deletion or hallucination if all hypotheses are wrong. Multimodal GER include the original audio as grounding to improve error correction.