Language Barriers: Evaluating Cross-Lingual Performance of CNN and Transformer Architectures for Speech Quality Estimation

TL;DR

This work investigates cross-lingual generalization of objective speech quality predictors trained on English by comparing CNN-based NISQA and Transformer-based AST on MOS across five languages. It uses third-order polynomial calibration per ITU-T P.1401 and evaluates five quality dimensions ( coloration, discontinuity, loudness, noise, MOS ) using PCC and RMSE. The AST model generally shows more stable cross-lingual performance, with Mandarin achieving the strongest correlations, while Swedish and Dutch pose notable challenges and discontinuity remains hard to predict. The findings highlight the need for balanced multilingual datasets and architecture-aware adaptations to improve cross-lingual robustness in automatic speech quality assessment.

Abstract

Objective speech quality models aim to predict human-perceived speech quality using automated methods. However, cross-lingual generalization remains a major challenge, as Mean Opinion Scores (MOS) vary across languages due to linguistic, perceptual, and dataset-specific differences. A model trained primarily on English data may struggle to generalize to languages with different phonetic, tonal, and prosodic characteristics, leading to inconsistencies in objective assessments. This study investigates the cross-lingual performance of two speech quality models: NISQA, a CNN-based model, and a Transformer-based Audio Spectrogram Transformer (AST) model. Both models were trained exclusively on English datasets containing over 49,000 speech samples and subsequently evaluated on speech in German, French, Mandarin, Swedish, and Dutch. We analyze model performance using Pearson Correlation Coefficient (PCC) and Root Mean Square Error (RMSE) across five speech quality dimensions: coloration, discontinuity, loudness, noise, and MOS. Our findings show that while AST achieves a more stable cross-lingual performance, both models exhibit noticeable biases. Notably, Mandarin speech quality predictions correlate highly with human MOS scores, whereas Swedish and Dutch present greater prediction challenges. Discontinuities remain difficult to model across all languages. These results highlight the need for more balanced multilingual datasets and architecture-specific adaptations to improve cross-lingual generalization.

Figures (4)

• Figure 1: NISQA model architecture
• Figure 2: AST-based model architecture
• Figure 3: MOS Prediction Scatterplot for Mandarin.
• Figure 4: MOS Prediction Scatterplot for Swedish.