Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Study on Zero-shot Non-intrusive Speech Assessment using Large Language Models

Ryandhimas E. Zezario, Sabato M. Siniscalchi, Hsin-Min Wang, Yu Tsao

TL;DR

The paper tackles zero-shot, non-intrusive speech assessment using large language models by comparing direct audio analysis with a GPT-Whisper pipeline that uses Whisper ASR and GPT-4o-based naturalness evaluation. GPT-4o alone shows limited capability for accurate speech assessment, while GPT-Whisper achieves moderate alignment with human quality (SRCC ≈ 0.4360) and intelligibility (SRCC ≈ 0.5485), and a strong correlation with Whisper CER (SRCC ≈ 0.7784). When benchmarked against DNSMOS, SpeechLMScore, MOS-SSL, and MTI-Net, GPT-Whisper excels in intelligibility and surpasses several supervised baselines in predicting Whisper CER. The findings validate GPT-Whisper as a promising zero-shot metric for speech quality, intelligibility, and CER, highlighting potential for reducing labeled data needs and enabling prompt-driven evaluation. Future work will explore richer prompt engineering and tighter integration with speech-processing workflows.

Abstract

This work investigates two strategies for zero-shot non-intrusive speech assessment leveraging large language models. First, we explore the audio analysis capabilities of GPT-4o. Second, we propose GPT-Whisper, which uses Whisper as an audio-to-text module and evaluates the naturalness of text via targeted prompt engineering. We evaluate the assessment metrics predicted by GPT-4o and GPT-Whisper, examining their correlation with human-based quality and intelligibility assessments and the character error rate (CER) of automatic speech recognition. Experimental results show that GPT-4o alone is less effective for audio analysis, while GPT-Whisper achieves higher prediction accuracy, has moderate correlation with speech quality and intelligibility, and has higher correlation with CER. Compared to SpeechLMScore and DNSMOS, GPT-Whisper excels in intelligibility metrics, but performs slightly worse than SpeechLMScore in quality estimation. Furthermore, GPT-Whisper outperforms supervised non-intrusive models MOS-SSL and MTI-Net in Spearman's rank correlation for CER of Whisper. These findings validate GPT-Whisper's potential for zero-shot speech assessment without requiring additional training data.

A Study on Zero-shot Non-intrusive Speech Assessment using Large Language Models

TL;DR

The paper tackles zero-shot, non-intrusive speech assessment using large language models by comparing direct audio analysis with a GPT-Whisper pipeline that uses Whisper ASR and GPT-4o-based naturalness evaluation. GPT-4o alone shows limited capability for accurate speech assessment, while GPT-Whisper achieves moderate alignment with human quality (SRCC ≈ 0.4360) and intelligibility (SRCC ≈ 0.5485), and a strong correlation with Whisper CER (SRCC ≈ 0.7784). When benchmarked against DNSMOS, SpeechLMScore, MOS-SSL, and MTI-Net, GPT-Whisper excels in intelligibility and surpasses several supervised baselines in predicting Whisper CER. The findings validate GPT-Whisper as a promising zero-shot metric for speech quality, intelligibility, and CER, highlighting potential for reducing labeled data needs and enabling prompt-driven evaluation. Future work will explore richer prompt engineering and tighter integration with speech-processing workflows.

Abstract

This work investigates two strategies for zero-shot non-intrusive speech assessment leveraging large language models. First, we explore the audio analysis capabilities of GPT-4o. Second, we propose GPT-Whisper, which uses Whisper as an audio-to-text module and evaluates the naturalness of text via targeted prompt engineering. We evaluate the assessment metrics predicted by GPT-4o and GPT-Whisper, examining their correlation with human-based quality and intelligibility assessments and the character error rate (CER) of automatic speech recognition. Experimental results show that GPT-4o alone is less effective for audio analysis, while GPT-Whisper achieves higher prediction accuracy, has moderate correlation with speech quality and intelligibility, and has higher correlation with CER. Compared to SpeechLMScore and DNSMOS, GPT-Whisper excels in intelligibility metrics, but performs slightly worse than SpeechLMScore in quality estimation. Furthermore, GPT-Whisper outperforms supervised non-intrusive models MOS-SSL and MTI-Net in Spearman's rank correlation for CER of Whisper. These findings validate GPT-Whisper's potential for zero-shot speech assessment without requiring additional training data.
Paper Structure (12 sections, 3 equations, 5 figures, 2 tables)

This paper contains 12 sections, 3 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. GPT-4o for speech assessment
  4. GPT-Whisper
  5. Experiments
  6. Experimental setup
  7. Experimental results of GPT-4o
  8. Experimental results of GPT-Whisper
  9. Correlation analysis
  10. Comparison with supervised methods
  11. Score distribution analysis
  12. Conclusions

Figures (5)

  • Figure 1: Zero-shot speech assessment with GPT-4o.
  • Figure 2: Zero-shot speech assessment with GPT-Whisper.
  • Figure 3: Example of GPT-4o used for speech assessment.
  • Figure 4: Example of GPT-Whisper used for speech assessment.
  • Figure 5: Distribution of predicted CER scores of GPT-Whisper and MTI-Net (Whisper) and Whisper's ground-truth CER scores.