Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PRiSM: Benchmarking Phone Realization in Speech Models

Shikhar Bharadwaj, Chin-Jou Li, Yoonjae Kim, Kwanghee Choi, Eunjung Yeo, Ryan Soh-Eun Shim, Hanyu Zhou, Brendon Boldt, Karen Rosero Jacome, Kalvin Chang, Darsh Agrawal, Keer Xu, Chao-Han Huck Yang, Jian Zhu, Shinji Watanabe, David R. Mortensen

TL;DR

PRiSM presents the first open-source benchmark for phone recognition that jointly evaluates intrinsic transcription accuracy and downstream utility across clinical, educational, and multilingual settings. It introduces Phonetic Feature Error Rate (PFER) for intrinsic evaluation and transcript/representation probes for extrinsic tasks, with a log-weighted aggregate for cross-task comparison. The study finds that broad, multilingual exposure and encoder-CTC architectures yield more robust phonetic capabilities, while Large Audio Language Models lag behind specialized PR models in phonetic perception. By releasing data, code, and recipes, PRiSM provides a scalable framework to advance multilingual speech models with robust phonetic understanding, guiding model selection and training for phonetic-accurate speech technologies.

Abstract

Phone recognition (PR) serves as the atomic interface for language-agnostic modeling for cross-lingual speech processing and phonetic analysis. Despite prolonged efforts in developing PR systems, current evaluations only measure surface-level transcription accuracy. We introduce PRiSM, the first open-source benchmark designed to expose blind spots in phonetic perception through intrinsic and extrinsic evaluation of PR systems. PRiSM standardizes transcription-based evaluation and assesses downstream utility in clinical, educational, and multilingual settings with transcription and representation probes. We find that diverse language exposure during training is key to PR performance, encoder-CTC models are the most stable, and specialized PR models still outperform Large Audio Language Models. PRiSM releases code, recipes, and datasets to move the field toward multilingual speech models with robust phonetic ability: https://github.com/changelinglab/prism.

PRiSM: Benchmarking Phone Realization in Speech Models

TL;DR

PRiSM presents the first open-source benchmark for phone recognition that jointly evaluates intrinsic transcription accuracy and downstream utility across clinical, educational, and multilingual settings. It introduces Phonetic Feature Error Rate (PFER) for intrinsic evaluation and transcript/representation probes for extrinsic tasks, with a log-weighted aggregate for cross-task comparison. The study finds that broad, multilingual exposure and encoder-CTC architectures yield more robust phonetic capabilities, while Large Audio Language Models lag behind specialized PR models in phonetic perception. By releasing data, code, and recipes, PRiSM provides a scalable framework to advance multilingual speech models with robust phonetic understanding, guiding model selection and training for phonetic-accurate speech technologies.

Abstract

Phone recognition (PR) serves as the atomic interface for language-agnostic modeling for cross-lingual speech processing and phonetic analysis. Despite prolonged efforts in developing PR systems, current evaluations only measure surface-level transcription accuracy. We introduce PRiSM, the first open-source benchmark designed to expose blind spots in phonetic perception through intrinsic and extrinsic evaluation of PR systems. PRiSM standardizes transcription-based evaluation and assesses downstream utility in clinical, educational, and multilingual settings with transcription and representation probes. We find that diverse language exposure during training is key to PR performance, encoder-CTC models are the most stable, and specialized PR models still outperform Large Audio Language Models. PRiSM releases code, recipes, and datasets to move the field toward multilingual speech models with robust phonetic ability: https://github.com/changelinglab/prism.
Paper Structure (42 sections, 2 equations, 5 figures, 6 tables)

This paper contains 42 sections, 2 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Phone Recognition Systems
  4. Phonetic information in PR systems
  5. Assessing phonetic/phonological ability
  6. Evaluation Framework of PRiSM
  7. Intrinsic: Core Capability
  8. Extrinsic: Downstream Utility
  9. Benchmarked Models
  10. Results and Discussion
  11. Intrinsic Evaluation
  12. Extrinsic Evaluations
  13. Analysis
  14. Phonotactics or the Acoustic Signal
  15. Zero-Shot Phonetic Inventory Induction
  16. ...and 27 more sections

Figures (5)

  • Figure 1: PRiSM is the first open-source benchmark for phone recognition systems, covering intrinsic and extrinsic evaluations, i.e., transcription task and downstream task performance.
  • Figure 2: PFER vs Phone masking rate. A PR model that relies only on acoustics should produce a horizontal line. Encoder-only models trained with CTC loss retain acoustic fidelity at high masking levels. See \ref{['sec:analysis_noise']}.
  • Figure 3: Precision and Recall scores of PR systems on phone inventory induction for unseen languages (\ref{['sec:analysis_phoneinv']}). CTC models trained with highly multilingual data are more stable.
  • Figure 4: Attribution map from Vaani vaani2025. Red supports and blue opposes correct geolocation. W2V2P-LV60 detects doubled phones (\ref{['sec:analysis_geo']}).
  • Figure 5: Normalized confusion matrices for Gemini 2.5 Flash on L1-eda (13 accent clusters). Rows denote true labels; columns denote predictions.