Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unveiling the Best Practices for Applying Speech Foundation Models to Speech Intelligibility Prediction for Hearing-Impaired People

Haoshuai Zhou, Boxuan Cao, Changgeng Mo, Linkai Li, Shan Xiang Wang

TL;DR

The paper tackles SIP-HI prediction using speech foundation models by adopting an adapter-based framework that freezes SFMs and trains lightweight heads. Through systematic ablations on encoder-layer selection, prediction-head architectures, and ensembling across five diverse SFMs, it reveals that single encoder layers often outperform all-layer fusion, temporal modeling in prediction heads is crucial, and ensembling strong SFMs yields robust gains over any single model. It also links SFM attributes to SIP-HI performance, showing ASR-focused models and newer architectures generally help, while training data volume has a nuanced effect. The work provides practical guidelines for designing SIP-HI predictors as more SFMs become available, emphasizing layer-wise evaluation, temporal-head design, and strategic ensembling to maximize performance.

Abstract

Speech foundation models (SFMs) have demonstrated strong performance across a variety of downstream tasks, including speech intelligibility prediction for hearing-impaired people (SIP-HI). However, optimizing SFMs for SIP-HI has been insufficiently explored. In this paper, we conduct a comprehensive study to identify key design factors affecting SIP-HI performance with 5 SFMs, focusing on encoder layer selection, prediction head architecture, and ensemble configurations. Our findings show that, contrary to traditional use-all-layers methods, selecting a single encoder layer yields better results. Additionally, temporal modeling is crucial for effective prediction heads. We also demonstrate that ensembling multiple SFMs improves performance, with stronger individual models providing greater benefit. Finally, we explore the relationship between key SFM attributes and their impact on SIP-HI performance. Our study offers practical insights into effectively adapting SFMs for speech intelligibility prediction for hearing-impaired populations.

Unveiling the Best Practices for Applying Speech Foundation Models to Speech Intelligibility Prediction for Hearing-Impaired People

TL;DR

The paper tackles SIP-HI prediction using speech foundation models by adopting an adapter-based framework that freezes SFMs and trains lightweight heads. Through systematic ablations on encoder-layer selection, prediction-head architectures, and ensembling across five diverse SFMs, it reveals that single encoder layers often outperform all-layer fusion, temporal modeling in prediction heads is crucial, and ensembling strong SFMs yields robust gains over any single model. It also links SFM attributes to SIP-HI performance, showing ASR-focused models and newer architectures generally help, while training data volume has a nuanced effect. The work provides practical guidelines for designing SIP-HI predictors as more SFMs become available, emphasizing layer-wise evaluation, temporal-head design, and strategic ensembling to maximize performance.

Abstract

Speech foundation models (SFMs) have demonstrated strong performance across a variety of downstream tasks, including speech intelligibility prediction for hearing-impaired people (SIP-HI). However, optimizing SFMs for SIP-HI has been insufficiently explored. In this paper, we conduct a comprehensive study to identify key design factors affecting SIP-HI performance with 5 SFMs, focusing on encoder layer selection, prediction head architecture, and ensemble configurations. Our findings show that, contrary to traditional use-all-layers methods, selecting a single encoder layer yields better results. Additionally, temporal modeling is crucial for effective prediction heads. We also demonstrate that ensembling multiple SFMs improves performance, with stronger individual models providing greater benefit. Finally, we explore the relationship between key SFM attributes and their impact on SIP-HI performance. Our study offers practical insights into effectively adapting SFMs for speech intelligibility prediction for hearing-impaired populations.
Paper Structure (14 sections, 4 figures, 4 tables)

This paper contains 14 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Experimental Setup
  3. Dataset
  4. Speech Foundation Models
  5. Encoder Feature Layer Selection
  6. Prediction Head Comparison
  7. Ensemble Different SFMs
  8. Training and Evaluation Details
  9. Results
  10. Self-Learned Layer Fusion Gives Suboptimal Solutions
  11. Temporal Modeling Is Important for Prediction Heads
  12. Stronger Individual SFMs Lead to Better Ensembles
  13. Discussion
  14. Conclusion

Figures (4)

  • Figure 1: Two prediction head structures used: WA-TGP and DT. Channel dimensions are omitted for illustration. The same color indicates shared weights.
  • Figure 2: SIP-HI performance across different encoder layer depths or using all layers for five SFMs. The top row shows DT results, while the bottom row shows WA-TGP results. Solid lines represent results with single encoder layers while dotted lines represent results using all encoder layers. Blue and red lines indicate the average RMSE and NCC scores across the three splits.
  • Figure 3: Violin and box plots showing the distribution of ensemble weights assigned to each SFM. The weights are output from a softmax layer, ensuring the sum in each ensemble equals 1.
  • Figure 4: Correlation between ranked SFM attributes and SIP-HI performance. Lower WER, larger data size, newer architecture, and more training tasks correspond to higher ranks when sorting.