Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How to Estimate Model Transferability of Pre-Trained Speech Models?

Zih-Ching Chen, Chao-Han Huck Yang, Bo Li, Yu Zhang, Nanxin Chen, Shuo-Yiin Chang, Rohit Prabhavalkar, Hung-yi Lee, Tara N. Sainath

TL;DR

This work tackles the problem of efficiently estimating how well pre-trained speech models transfer to downstream tasks without costly fine-tuning. It introduces a score-based framework combining a TIH-enabled optimal-transport latent-space distance and a LogME-inspired Bayesian evidence approach to assess transferability, applicable to both supervised and self-supervised speech models. Key contributions include TIH-integrated SWD and LogME formulations for speech, comprehensive layer-wise and model-wise evaluations, and strong correlations with actual fine-tuning performance, all while reducing computational resources. The proposed methods offer practical guidance for selecting PSMs and tuning strategies in resource-constrained settings, with code provided for reproducibility.

Abstract

In this work, we introduce a "score-based assessment" framework for estimating the transferability of pre-trained speech models (PSMs) for fine-tuning target tasks. We leverage upon two representation theories, Bayesian likelihood estimation and optimal transport, to generate rank scores for the PSM candidates using the extracted representations. Our framework efficiently computes transferability scores without actual fine-tuning of candidate models or layers by making a temporal independent hypothesis. We evaluate some popular supervised speech models (e.g., Conformer RNN-Transducer) and self-supervised speech models (e.g., HuBERT) in cross-layer and cross-model settings using public data. Experimental results show a high Spearman's rank correlation and low $p$-value between our estimation framework and fine-tuning ground truth. Our proposed transferability framework requires less computational time and resources, making it a resource-saving and time-efficient approach for tuning speech foundation models.

How to Estimate Model Transferability of Pre-Trained Speech Models?

TL;DR

This work tackles the problem of efficiently estimating how well pre-trained speech models transfer to downstream tasks without costly fine-tuning. It introduces a score-based framework combining a TIH-enabled optimal-transport latent-space distance and a LogME-inspired Bayesian evidence approach to assess transferability, applicable to both supervised and self-supervised speech models. Key contributions include TIH-integrated SWD and LogME formulations for speech, comprehensive layer-wise and model-wise evaluations, and strong correlations with actual fine-tuning performance, all while reducing computational resources. The proposed methods offer practical guidance for selecting PSMs and tuning strategies in resource-constrained settings, with code provided for reproducibility.

Abstract

In this work, we introduce a "score-based assessment" framework for estimating the transferability of pre-trained speech models (PSMs) for fine-tuning target tasks. We leverage upon two representation theories, Bayesian likelihood estimation and optimal transport, to generate rank scores for the PSM candidates using the extracted representations. Our framework efficiently computes transferability scores without actual fine-tuning of candidate models or layers by making a temporal independent hypothesis. We evaluate some popular supervised speech models (e.g., Conformer RNN-Transducer) and self-supervised speech models (e.g., HuBERT) in cross-layer and cross-model settings using public data. Experimental results show a high Spearman's rank correlation and low -value between our estimation framework and fine-tuning ground truth. Our proposed transferability framework requires less computational time and resources, making it a resource-saving and time-efficient approach for tuning speech foundation models.
Paper Structure (12 sections, 4 equations, 2 figures, 4 tables)

This paper contains 12 sections, 4 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Transferability for Speech Models
  3. Modeling Temporal Independent Hypothesis (TIH)
  4. Optimal Transport by Sliced Wasserstein Distance
  5. Transferability Estimation by Likelihood
  6. Experiments
  7. Experimental setup
  8. Study 1: Layer-Wise Exploration
  9. Study 2: Model-Wise Exploration
  10. Pre-trained speech model tuned on classification
  11. Different pre-train data on same neural architecture
  12. Conclusion

Figures (2)

  • Figure 1: A step-by-step illustration of the proposed framework for providing scores for assessing the best pre-trained model and the best layer for transfer learning with speech data.
  • Figure 2: Illustration of the two approaches for estimating transferability in speech processing tasks: optimal transport (Section \ref{['2:2']}) and maximum evidence (Section \ref{['2:3']}). The transferability metric in \ref{['2:2']} is SWD yang2021voice2series, while in \ref{['2:3']}, we use LogME ($\log p(\boldsymbol{y}|F) / n$), where $p(\boldsymbol{y}|F)=\int p(w)p(\boldsymbol{y}|F,w)dw$, to assess transferability.