Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enabling Auditory Large Language Models for Automatic Speech Quality Evaluation

Siyin Wang, Wenyi Yu, Yudong Yang, Changli Tang, Yixuan Li, Jimin Zhuang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yuxuan Wang, Chao Zhang

TL;DR

This work investigates the use of auditory large language models (LLMs) for automatic speech quality evaluation, aiming to unify MOS, SIM, A/B testing, and natural-language descriptions within a single multitask framework. By finetuning open-source auditory LLMs (e.g., SALMONN, Qwen-Audio, Qwen2-Audio) with task-specific prompts and LoRA-based adapters, the authors demonstrate competitive MOS and SIM prediction on NISQA, BVCC, SOMOS, and VoxSim datasets, along with promising A/B testing and descriptive outputs. The approach outperforms several task-specific baselines and offers interpretable NL assessments of noisiness, distortion, and overall quality, highlighting the potential of auditory LLMs for comprehensive, scalable speech quality evaluation. The results suggest a practical impact for efficient evaluation of TTS systems and data curation, with future work pointing to improved NL-description diversity and broader dataset coverage.

Abstract

Speech quality assessment typically requires evaluating audio from multiple aspects, such as mean opinion score (MOS) and speaker similarity (SIM) \etc., which can be challenging to cover using one small model designed for a single task. In this paper, we propose leveraging recently introduced auditory large language models (LLMs) for automatic speech quality assessment. By employing task-specific prompts, auditory LLMs are finetuned to predict MOS, SIM and A/B testing results, which are commonly used for evaluating text-to-speech systems. Additionally, the finetuned auditory LLM is able to generate natural language descriptions assessing aspects like noisiness, distortion, discontinuity, and overall quality, providing more interpretable outputs. Extensive experiments have been performed on the NISQA, BVCC, SOMOS and VoxSim speech quality datasets, using open-source auditory LLMs such as SALMONN, Qwen-Audio, and Qwen2-Audio. For the natural language descriptions task, a commercial model Google Gemini 1.5 Pro is also evaluated. The results demonstrate that auditory LLMs achieve competitive performance compared to state-of-the-art task-specific small models in predicting MOS and SIM, while also delivering promising results in A/B testing and natural language descriptions. Our data processing scripts and finetuned model checkpoints can be found at https://github.com/bytedance/SALMONN.

Enabling Auditory Large Language Models for Automatic Speech Quality Evaluation

TL;DR

This work investigates the use of auditory large language models (LLMs) for automatic speech quality evaluation, aiming to unify MOS, SIM, A/B testing, and natural-language descriptions within a single multitask framework. By finetuning open-source auditory LLMs (e.g., SALMONN, Qwen-Audio, Qwen2-Audio) with task-specific prompts and LoRA-based adapters, the authors demonstrate competitive MOS and SIM prediction on NISQA, BVCC, SOMOS, and VoxSim datasets, along with promising A/B testing and descriptive outputs. The approach outperforms several task-specific baselines and offers interpretable NL assessments of noisiness, distortion, and overall quality, highlighting the potential of auditory LLMs for comprehensive, scalable speech quality evaluation. The results suggest a practical impact for efficient evaluation of TTS systems and data curation, with future work pointing to improved NL-description diversity and broader dataset coverage.

Abstract

Speech quality assessment typically requires evaluating audio from multiple aspects, such as mean opinion score (MOS) and speaker similarity (SIM) \etc., which can be challenging to cover using one small model designed for a single task. In this paper, we propose leveraging recently introduced auditory large language models (LLMs) for automatic speech quality assessment. By employing task-specific prompts, auditory LLMs are finetuned to predict MOS, SIM and A/B testing results, which are commonly used for evaluating text-to-speech systems. Additionally, the finetuned auditory LLM is able to generate natural language descriptions assessing aspects like noisiness, distortion, discontinuity, and overall quality, providing more interpretable outputs. Extensive experiments have been performed on the NISQA, BVCC, SOMOS and VoxSim speech quality datasets, using open-source auditory LLMs such as SALMONN, Qwen-Audio, and Qwen2-Audio. For the natural language descriptions task, a commercial model Google Gemini 1.5 Pro is also evaluated. The results demonstrate that auditory LLMs achieve competitive performance compared to state-of-the-art task-specific small models in predicting MOS and SIM, while also delivering promising results in A/B testing and natural language descriptions. Our data processing scripts and finetuned model checkpoints can be found at https://github.com/bytedance/SALMONN.
Paper Structure (18 sections, 1 figure, 5 tables)

This paper contains 18 sections, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Auditory LLM for Diverse Comprehension Tasks
  4. Automatic Speech Quality Evaluation
  5. Methodology
  6. Auditory LLMs for Audio Perception and Understanding
  7. Speech Quality Evaluation with Auditory LLM
  8. Experiment Setup
  9. Datasets
  10. Models
  11. Baselines
  12. Evaluation Criteria
  13. Results
  14. MOS Prediction
  15. SIM Prediction
  16. ...and 3 more sections

Figures (1)

  • Figure 1: Sketchmaps of the speech quality assessment models. (a) The speech SSL model baseline. The SSL model and downstream models are fully finetuned using the speech quality assessment dataset, involving MOS and SIM prediction. (b) Speech quality assessment auditory LLM. The auditory LLM is finetuned with LoRA on the speech quality assessment dataset. Task-specific prompts are used to enable auditory LLMs to perform different speech assessment tasks, including MOS/SIM prediction, speech quality A/B testing and natural language descriptions.