Zero-Shot Speech LLMs for Multi-Aspect Evaluation of L2 Speech: Challenges and Opportunities

TL;DR

This work investigates zero-shot, multi-aspect pronunciation assessment using the instruction-tuned speech LLM Qwen2-Audio-7B-Instruct on Speechocean762. By pairing a Whisper-based audio encoder with a Qwen-7B decoder and a multimodal prompting strategy, the study demonstrates coarse-grained alignment with human ratings across four rubrics (accuracy, fluency, prosody, completeness) without task-specific fine-tuning, achieving reasonable agreement within $\pm$2 points, especially for higher-quality speech. However, the model exhibits an overestimation bias for low-quality utterances and limited precision in error detection, with completeness scoring particularly problematic likely due to rubric ambiguity. The results suggest strong potential for scalable pronunciation evaluation and CAPT prototyping, while pointing to future work in calibration, prompt optimization, and phonetic-level integration to enable more interpretable and fine-grained feedback.

Abstract

An accurate assessment of L2 English pronunciation is crucial for language learning, as it provides personalized feedback and ensures a fair evaluation of individual progress. However, automated scoring remains challenging due to the complexity of sentence-level fluency, prosody, and completeness. This paper evaluates the zero-shot performance of Qwen2-Audio-7B-Instruct, an instruction-tuned speech-LLM, on 5,000 Speechocean762 utterances. The model generates rubric-aligned scores for accuracy, fluency, prosody, and completeness, showing strong agreement with human ratings within +-2 tolerance, especially for high-quality speech. However, it tends to overpredict low-quality speech scores and lacks precision in error detection. These findings demonstrate the strong potential of speech LLMs in scalable pronunciation assessment and suggest future improvements through enhanced prompting, calibration, and phonetic integration to advance Computer-Assisted Pronunciation Training.

Figures (1)

• Figure 1: Score distributions for GT and model predictions across all rubrics. GT bars are solid and opaque, while model predictions use lighter colors with striped hatch patterns. Each rubric is shown using a distinct color with grouped bars.