Extending Whisper with prompt tuning to target-speaker ASR

TL;DR

Target-speaker ASR from overlapped speech is challenging for single-talker models. The paper develops a parameter-efficient approach that extends Whisper to TS-ASR using prompt tuning, deep prompting, and reparameterization. It achieves competitive performance with only about 1% of task-specific parameters, while preserving Whisper's inverse text normalization and timestamping abilities. This work demonstrates a practical path to adapting large foundation models to multi-talker TS-ASR with minimal fine-tuning cost.

Abstract

Target-speaker automatic speech recognition (ASR) aims to transcribe the desired speech of a target speaker from multi-talker overlapped utterances. Most of the existing target-speaker ASR (TS-ASR) methods involve either training from scratch or fully fine-tuning a pre-trained model, leading to significant training costs and becoming inapplicable to large foundation models. This work leverages prompt tuning, a parameter-efficient fine-tuning approach, to extend Whisper, a large-scale single-talker ASR model, to TS-ASR. Variants of prompt tuning approaches along with their configurations are explored and optimized for TS-ASR.Experimental results show that prompt tuning can achieve performance comparable to state-of-the-art full training approaches while only requiring about 1\% of task-specific model parameters. Notably, the original Whisper's features, such as inverse text normalization and timestamp tagging, are retained in target-speaker ASR, keeping the generated transcriptions natural and informative.

Figures (1)

• Figure 1: Overview of proposed prompting framework. Modules with solid-line borders are included in the baseline configuration, while modules with dashed-line borders are optional.