Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UniWhisper: Efficient Continual Multi-task Training for Robust Universal Audio Representation

Yuxuan Chen, Peize He, Haoyuan Xu, Junzi Zhang

TL;DR

UniWhisper is proposed, an efficient continual multi-task training framework that casts heterogeneous audio tasks into a unified instruction and answer format that enables standard next-token training without task-specific heads and losses.

Abstract

A universal audio representation should capture fine-grained speech cues and high-level semantics for environmental sounds and music in a single encoder. Existing encoders often excel in one domain but degrade in others. We propose UniWhisper, an efficient continual multi-task training framework that casts heterogeneous audio tasks into a unified instruction and answer format. This enables standard next-token training without task-specific heads and losses. We train it on 38k hours of public audio and assess the encoder using shallow MLP probes and k-nearest neighbors (kNN) on 20 tasks spanning speech, environmental sound, and music. UniWhisper reaches normalized weighted averages of 0.81 with MLP probes and 0.61 with kNN, compared to 0.64 and 0.46 for Whisper, while retaining strong speech performance.

UniWhisper: Efficient Continual Multi-task Training for Robust Universal Audio Representation

TL;DR

UniWhisper is proposed, an efficient continual multi-task training framework that casts heterogeneous audio tasks into a unified instruction and answer format that enables standard next-token training without task-specific heads and losses.

Abstract

A universal audio representation should capture fine-grained speech cues and high-level semantics for environmental sounds and music in a single encoder. Existing encoders often excel in one domain but degrade in others. We propose UniWhisper, an efficient continual multi-task training framework that casts heterogeneous audio tasks into a unified instruction and answer format. This enables standard next-token training without task-specific heads and losses. We train it on 38k hours of public audio and assess the encoder using shallow MLP probes and k-nearest neighbors (kNN) on 20 tasks spanning speech, environmental sound, and music. UniWhisper reaches normalized weighted averages of 0.81 with MLP probes and 0.61 with kNN, compared to 0.64 and 0.46 for Whisper, while retaining strong speech performance.
Paper Structure (14 sections, 3 equations, 1 figure, 4 tables)

This paper contains 14 sections, 3 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Backbone and initialization
  4. Unified instruction style multitask training
  5. Experimental Setup
  6. Datasets
  7. Training details
  8. Evaluation protocols
  9. Results
  10. MLP results
  11. kNN results
  12. Ablation
  13. Conclusion
  14. Generative AI Use Disclosure

Figures (1)

  • Figure 1: Normalized per-task performance of UniWhisper on our 20-task extended HEAREval spanning speech, environmental sound, and music. Full comparisons are reported in Table \ref{['tab:full_results']}.