Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MT2KD: Towards A General-Purpose Encoder for Speech, Speaker, and Audio Events

Xiaoyu Yang, Qiujia Li, Chao Zhang, Phil Woodland

TL;DR

MT2KD introduces a two-stage framework to build a general-purpose audio encoder that jointly supports ASR, AT, and SV. Stage 1 uses multi-teacher knowledge distillation on unlabelled data to align the feature spaces of three task-specific teachers, producing a strong initial student encoder. Stage 2 performs multi-task supervised fine-tuning, initializing from the KD pre-trained model, yielding competitive WER, mAP, and EER with far fewer parameters than task-specific models. The approach demonstrates that careful cross-task alignment via KD and staged fine-tuning can overcome interference between tasks and deliver a practical, efficient foundation for multi-task speech/audio processing.

Abstract

With the advances in deep learning, the performance of end-to-end (E2E) single-task models for speech and audio processing has been constantly improving. However, it is still challenging to build a general-purpose model with high performance on multiple tasks, since different speech and audio processing tasks usually require different training data, input features, or model architectures to achieve optimal performance. In this work, MT2KD, a novel two-stage multi-task learning framework is proposed to build a general-purpose speech and audio encoder that jointly performs three fundamental tasks: automatic speech recognition (ASR), audio tagging (AT) and speaker verification (SV). In the first stage, multi-teacher knowledge distillation (KD) is applied to align the feature spaces of three single-task high-performance teacher encoders into a single student encoder using the same unlabelled data. In the second stage, multi-task supervised fine-tuning is carried out by initialising the model from the first stage and training on the separate labelled data of each single task. Experiments demonstrate that the proposed multi-task training pipeline significantly outperforms a baseline model trained with multi-task learning from scratch. The final system achieves good performance on ASR, AT and SV: with less than 4% relative word-error-rate increase on ASR, only 1.9 lower mean averaged precision on AT and 0.23% absolute higher equal error rate on SV compared to the best-performing single-task encoders, using only a 66M total model parameters.

MT2KD: Towards A General-Purpose Encoder for Speech, Speaker, and Audio Events

TL;DR

MT2KD introduces a two-stage framework to build a general-purpose audio encoder that jointly supports ASR, AT, and SV. Stage 1 uses multi-teacher knowledge distillation on unlabelled data to align the feature spaces of three task-specific teachers, producing a strong initial student encoder. Stage 2 performs multi-task supervised fine-tuning, initializing from the KD pre-trained model, yielding competitive WER, mAP, and EER with far fewer parameters than task-specific models. The approach demonstrates that careful cross-task alignment via KD and staged fine-tuning can overcome interference between tasks and deliver a practical, efficient foundation for multi-task speech/audio processing.

Abstract

With the advances in deep learning, the performance of end-to-end (E2E) single-task models for speech and audio processing has been constantly improving. However, it is still challenging to build a general-purpose model with high performance on multiple tasks, since different speech and audio processing tasks usually require different training data, input features, or model architectures to achieve optimal performance. In this work, MT2KD, a novel two-stage multi-task learning framework is proposed to build a general-purpose speech and audio encoder that jointly performs three fundamental tasks: automatic speech recognition (ASR), audio tagging (AT) and speaker verification (SV). In the first stage, multi-teacher knowledge distillation (KD) is applied to align the feature spaces of three single-task high-performance teacher encoders into a single student encoder using the same unlabelled data. In the second stage, multi-task supervised fine-tuning is carried out by initialising the model from the first stage and training on the separate labelled data of each single task. Experiments demonstrate that the proposed multi-task training pipeline significantly outperforms a baseline model trained with multi-task learning from scratch. The final system achieves good performance on ASR, AT and SV: with less than 4% relative word-error-rate increase on ASR, only 1.9 lower mean averaged precision on AT and 0.23% absolute higher equal error rate on SV compared to the best-performing single-task encoders, using only a 66M total model parameters.
Paper Structure (28 sections, 7 equations, 1 figure, 12 tables)

This paper contains 28 sections, 7 equations, 1 figure, 12 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Automatic Speech Recognition
  4. Audio Tagging
  5. Speaker Verification
  6. Multi-task Learning of Speech and Audio Processing
  7. Knowledge Distillation in Speech and Audio Processing
  8. General-purpose Audio Encoder
  9. Motivation
  10. Naive Multi-task Training
  11. Multi-teacher Multi-task KD Pre-training
  12. Multi-task Fine-tuning
  13. Experimental Setup
  14. Teacher Models
  15. Student Model
  16. ...and 13 more sections

Figures (1)

  • Figure 1: The multi-teacher multi-task KD pre-training framework. The feature extraction modules for each teacher are included in the teacher models. The teacher models are frozen all the time. The modules in the dashed box belong to the student model and the convolution subsampling module is emitted.