VoiceAgentBench: Are Voice Assistants ready for agentic tasks?

TL;DR

VoiceAgentBench introduces a comprehensive, multilingual benchmark for evaluating speech-based agentic capabilities, capturing single- and multi-tool invocation, multi-turn dialogues, interdependent tool calls, and safety judgments across English and multiple Indic languages. By coupling 5,500+ synthetic queries with diversity-aware TTS and culturally grounded Indian-context prompts, the benchmark reveals substantial gaps in current SpeechLMs’ ability to reason, plan, and safely orchestrate tools compared to ASR-LLM pipelines, especially in Indic languages. The work systematically defines evaluation metrics (Tool Selection, Tool Call Structure, Parameter Filling, and Refusal Rate) to quantify these capabilities and robustness, and shows that end-to-end SpeechLMs face notable challenges as task complexity grows. Overall, VAB provides a rigorous framework and dataset to drive development of low-latency, culturally aware, and safe voice agents capable of complex, real-world workflows.

Abstract

Large-scale Speech Language Models (SpeechLMs) have enabled voice assistants capable of understanding natural spoken queries and performing complex tasks. However, existing speech benchmarks primarily focus on isolated capabilities such as transcription, or question-answering, and do not systematically evaluate agentic scenarios encompassing multilingual and cultural understanding, as well as adversarial robustness. To address this, we introduce VoiceAgentBench, a comprehensive benchmark designed to evaluate SpeechLMs in realistic spoken agentic settings. It comprises over 5,500 synthetic spoken queries, including dialogues grounded in Indian context, covering single-tool invocations, multi-tool workflows, multi-turn interactions, and safety evaluations. The benchmark supports English, Hindi, and 5 other Indian languages, reflecting real-world linguistic and cultural diversity. We simulate speaker variability using a novel sampling algorithm that selects audios for TTS voice conversion based on its speaker embeddings, maximizing acoustic and speaker diversity. Our evaluation measures tool selection accuracy, structural consistency, and the correctness of tool invocations, including adversarial robustness. Our experiments reveal significant gaps in contextual tool orchestration tasks, Indic generalization, and adversarial robustness, exposing critical limitations of current SpeechLMs.

Figures (9)

• Figure 1: Overview of the different agentic task categories in VoiceAgentBench, illustrating representative examples for each type of tool interaction, including single tool invocation, parallel and sequential tool use, multi-turn dialogue handling, and safety against harmful requests. The benchmark also supports multilingual capabilities, particularly for Indic languages.
• Figure 2: Pipeline for constructing VoiceAgentBench. We begin with seed tools, dialogues, and custom APIs for diverse agentic tasks. Indic grounding and TTS engine generate culturally contextualized speech queries, while diversity sampling ensures coverage across accents, and speakers. The final benchmark pairs user speech with tool context and model instructions.
• Figure 3: Comparison of diversity sampling methods using audio embeddings. We report the mean pairwise distance of the selected samples and visualize their distribution with t-SNE plots.
• Figure 4: Comparison of Model performance with and without refusal prompts for Safety tasks.
• Figure 5: Comparison of Model performance with and without hint in the queries for Safety tasks.