SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Models

TL;DR

SpeechIQ (SIQ) offers a cognition-inspired, three-level evaluation of LLM_Voice across Remember, Understand, and Apply, addressing the shortcomings of WER by incorporating semantic preservation and task-execution metrics. The framework enables unified comparisons across cascaded ASR+LLM, GER-enhanced cascaded systems, and end-to-end multimodal models, with a final SIQ score computed via discrimination weights, standardization, and dynamic weighting. Key findings show cascaded approaches generally outperform end-to-end on SIQ, GER improves performance for ASR+LLM, and the unanswerable set helps detect hallucinations, while human alignment supports SIQ as a robust benchmark. The work outlines practical implications for robust voice understanding and highlights limitations, scaling considerations, and ethical aspects, pointing to future work on higher Bloom levels and scaling-normalized evaluations.

Abstract

We introduce Speech-based Intelligence Quotient (SIQ) as a new form of human cognition-inspired evaluation pipeline for voice understanding large language models, LLM Voice, designed to assess their voice understanding ability. Moving beyond popular voice understanding metrics such as word error rate (WER), SIQ examines LLM Voice across three cognitive levels motivated by Bloom's Taxonomy: (1) Remembering (i.e., WER for verbatim accuracy); (2) Understanding (i.e., similarity of LLM's interpretations); and (3) Application (i.e., QA accuracy for simulating downstream tasks). We demonstrate that SIQ not only quantifies voice understanding abilities but also provides unified comparisons between cascaded methods (e.g., ASR LLM) and end-to-end models, identifies annotation errors in existing benchmarks, and detects hallucinations in LLM Voice. Our framework represents a first-of-its-kind intelligence examination that bridges cognitive principles with voice-oriented benchmarks, while exposing overlooked challenges in multi-modal training. Our code and data will be open source to encourage future studies.

Figures (19)

• Figure 1: An overview of cognitive levels in voice understanding large language model systems related to the first three foundational hierarchies of Bloom's Taxonomy. We design a corresponding examination to measure SpeechIQ as a detailed pipeline in Figure \ref{['result: method']}.
• Figure 2: Overview of Speech IQ (SIQ) test. We compared cognition-inspired categories of LLM$_\text{Voice}$ represented in three rows, with the right columns denoting the SIQ sub-tests. The truth reference will be used in the sub-tests.
• Figure 3: Raw remember scores. All cascaded models outperform end-to-end models including Gemini-1.5.
• Figure 4: Raw understand scores. Gemini-1.5-flash shows the best semantic capturing.
• Figure 5: Raw apply scores. End-to-end-large models perform best while smaller ones perform worst.