FOCAL: A Novel Benchmarking Technique for Multi-modal Agents
Aditya Choudhary, Anupam Purwar
TL;DR
FOCAL tackles the lack of standardized end-to-end benchmarking for multi-modal voice-and-text agents by introducing a pipeline that models user interactions with a Human-Simulator, processes inputs via TTS/ASR, and evaluates rich metrics that capture both reasoning and semantic quality. The framework defines a novel REST-inspired scoring scheme (R-E-S-T) and a suite of vocal, linguistic, and tool-use metrics to diagnose error propagation across components. Demonstrated on a RAG-based shopping agent, FOCAL provides quantitative end-to-end assessment, component-level diagnostics, and a live demo pipeline to facilitate reproducibility and broader adoption. This work enables more reliable development of voice-enabled assistants with robust, coherent, and consistent behavior in multi-turn conversations.
Abstract
With the recent advancements in reasoning capabilities, tool calling using MCP servers and Audio Language Models (ALMs), development and integration of multi-modal agents (with voice and text support) has come to the industry forefront. Cascading pipelines for voice agents still play a central role in the industry owing to their superior reasoning capabilities facilitated by LLMs. Although, cascading pipelines often present error propagation through the pipeline. We propose a framework, FOCAL to benchmark end-to-end reasoning, component-wise error propagation and error analysis for automated as well as human-assisted testing of multi-modal agents (voice to voice + text input). We also share two novel metrics viz. Reasoning and Semantic scores to evaluate efficacy of the agent in having meaningful conversations in voice mode.