PolySpeech: Exploring Unified Multitask Speech Models for Competitiveness with Single-task Models
Runyan Yang, Huibao Yang, Xiqing Zhang, Tiantian Ye, Ying Liu, Yingying Gao, Shilei Zhang, Chao Deng, Junlan Feng
TL;DR
PolySpeech tackles the challenge of unifying multiple speech tasks within a single multitask model by employing a decoder-only multimodal Transformer that leverages semantic speech representations. It introduces Semantic Speech Embedding Tokenization (SSET) and a non-autoregressive waveform decoder with speaker prompts and a high-fidelity vocoder to enable high-quality, speaker-conditioned TTS. Experimental results show that multitask optimization can achieve performance comparable to single-task training and can surpass it for certain tasks, while semantic tokens provide better generalization for tasks like LID. The work demonstrates the practicality of unified multitask speech models and lays groundwork for extending to additional tasks and languages, with implications for efficiency and scalability in speech AI systems.
Abstract
Recently, there have been attempts to integrate various speech processing tasks into a unified model. However, few previous works directly demonstrated that joint optimization of diverse tasks in multitask speech models has positive influence on the performance of individual tasks. In this paper we present a multitask speech model -- PolySpeech, which supports speech recognition, speech synthesis, and two speech classification tasks. PolySpeech takes multi-modal language model as its core structure and uses semantic representations as speech inputs. We introduce semantic speech embedding tokenization and speech reconstruction methods to PolySpeech, enabling efficient generation of high-quality speech for any given speaker. PolySpeech shows competitiveness across various tasks compared to single-task models. In our experiments, multitask optimization achieves performance comparable to single-task optimization and is especially beneficial for specific tasks.