SpeechLLM-as-Judges: Towards General and Interpretable Speech Quality Evaluation

TL;DR

This work tackles the interpretability and generalization gap in speech quality evaluation by introducing SpeechLLM-as-Judges, where large language models perform structured, explanation-based evaluations across multiple tasks and languages. It introduces SpeechEval, a large-scale multilingual dataset with 32,207 clips and 128,754 annotations covering quality assessment, comparisons, improvement suggestions, and deepfake detection, and proposes SQ-LLM, a speech-quality-aware LLM trained with Chain-of-Thought reasoning and Generalized Policy Gradient Optimization. Experiments show SQ-LLM achieves strong, human-aligned performance across tasks and languages, outperforming task-specific and zero-shot baselines while providing interpretable rationales. The work demonstrates the potential of a unified, interpretable evaluation framework for advancing perceptual speech quality in multilingual, real-world settings, with resources to be open-sourced for broader impact.

Abstract

Generative speech technologies are progressing rapidly, but evaluating the perceptual quality of synthetic speech remains a core challenge. Existing methods typically rely on scalar scores or binary decisions, which lack interpretability and generalization across tasks and languages. We present SpeechLLM-as-Judges, a new paradigm for enabling large language models (LLMs) to conduct structured and explanation-based speech quality evaluation. To support this direction, we introduce SpeechEval, a large-scale dataset containing 32,207 multilingual speech clips and 128,754 annotations spanning four tasks: quality assessment, pairwise comparison, improvement suggestion, and deepfake detection. Based on this resource, we develop SQ-LLM, a speech-quality-aware LLM trained with chain-of-thought reasoning and reward optimization to improve capability. Experimental results show that SQ-LLM delivers strong performance across tasks and languages, revealing the potential of this paradigm for advancing speech quality evaluation. Relevant resources will be open-sourced.

Figures (21)

• Figure 1: Example interactions showcasing the core capabilities of SpeechLLM-as-Judges. The model supports: speech quality assessment, comparison, improvement suggestion, and deepfake detection.
• Figure 2: Per-task statistics across four languages, highlighting both the number of audio items (light color) and annotations (solid color).
• Figure 3: Overview of the SpeechEval data construction process, including data collection (left), task-specific annotation protocols (middle), and a human-in-the-loop annotation workflow with LLM assistance (right).
• Figure 4: Overview of SQ-LLM training. Stage I uses instruction tuning with dimension-wise CoT reasoning. Stage II applies multi-aspect feedback for refinement.
• Figure 5: Performance of SQ-LLM across tasks and languages. Left y-axis reports LLM Scores; right y-axis shows accuracy for Deepfake Detection.