Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SpeechComposer: Unifying Multiple Speech Tasks with Prompt Composition

Yihan Wu, Soumi Maiti, Yifan Peng, Wangyou Zhang, Chenda Li, Yuyue Wang, Xihua Wang, Shinji Watanabe, Ruihua Song

TL;DR

SpeechComposer introduces a decoder-only framework that unifies multiple speech tasks by composing a fixed set of prompt tokens for four primary tasks: SpeechLM, TextLM, ASR, and TTS. By treating all speech tasks as composites of these primaries and enabling extensions like voice conversion and speech enhancement through enrollment speech, the model shares knowledge across tasks without task-specific prompt tokens. Empirical results show competitive or superior performance on primary and composite tasks relative to expert and language-based baselines, with benefits amplified by larger models and more training tasks, and demonstrated zero-shot transfer to unseen composites. The approach offers a scalable, reproducible path toward cross-task sharing in speech, supported by public datasets, a standardized data format, and an open-source training/inference workflow.

Abstract

Recent advancements in language models have significantly enhanced performance in multiple speech-related tasks. Existing speech language models typically utilize task-dependent prompt tokens to unify various speech tasks in a single model. However, this design omits the intrinsic connections between different speech tasks, which can potentially boost the performance of each task. In this work, we propose a novel decoder-only speech language model, SpeechComposer, that can unify common speech tasks by composing a fixed set of prompt tokens. Built upon four primary tasks -- speech synthesis, speech recognition, speech language modeling, and text language modeling -- SpeechComposer can easily extend to more speech tasks via compositions of well-designed prompt tokens, like voice conversion and speech enhancement. The unification of prompt tokens also makes it possible for knowledge sharing among different speech tasks in a more structured manner. Experimental results demonstrate that our proposed SpeechComposer can improve the performance of both primary tasks and composite tasks, showing the effectiveness of the shared prompt tokens. Remarkably, the unified decoder-only model achieves a comparable and even better performance than the baselines which are expert models designed for single tasks.

SpeechComposer: Unifying Multiple Speech Tasks with Prompt Composition

TL;DR

SpeechComposer introduces a decoder-only framework that unifies multiple speech tasks by composing a fixed set of prompt tokens for four primary tasks: SpeechLM, TextLM, ASR, and TTS. By treating all speech tasks as composites of these primaries and enabling extensions like voice conversion and speech enhancement through enrollment speech, the model shares knowledge across tasks without task-specific prompt tokens. Empirical results show competitive or superior performance on primary and composite tasks relative to expert and language-based baselines, with benefits amplified by larger models and more training tasks, and demonstrated zero-shot transfer to unseen composites. The approach offers a scalable, reproducible path toward cross-task sharing in speech, supported by public datasets, a standardized data format, and an open-source training/inference workflow.

Abstract

Recent advancements in language models have significantly enhanced performance in multiple speech-related tasks. Existing speech language models typically utilize task-dependent prompt tokens to unify various speech tasks in a single model. However, this design omits the intrinsic connections between different speech tasks, which can potentially boost the performance of each task. In this work, we propose a novel decoder-only speech language model, SpeechComposer, that can unify common speech tasks by composing a fixed set of prompt tokens. Built upon four primary tasks -- speech synthesis, speech recognition, speech language modeling, and text language modeling -- SpeechComposer can easily extend to more speech tasks via compositions of well-designed prompt tokens, like voice conversion and speech enhancement. The unification of prompt tokens also makes it possible for knowledge sharing among different speech tasks in a more structured manner. Experimental results demonstrate that our proposed SpeechComposer can improve the performance of both primary tasks and composite tasks, showing the effectiveness of the shared prompt tokens. Remarkably, the unified decoder-only model achieves a comparable and even better performance than the baselines which are expert models designed for single tasks.
Paper Structure (48 sections, 2 equations, 2 figures, 7 tables)

This paper contains 48 sections, 2 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Speech Language Models
  4. Composite Speech Task Frameworks
  5. Proposed Method
  6. Data Format
  7. Model architectures
  8. SpeechComposer.
  9. Speech tokenizer.
  10. Speech token decoder.
  11. Training and Inference
  12. Experiment Setup
  13. Model configurations
  14. Datasets
  15. SpeechLM.
  16. ...and 33 more sections

Figures (2)

  • Figure 1: The comparison of SpeechComposer with other multi-task speech models when performing $N$ tasks. (a) shows a cascaded model that requires different sub-models for each task. (b) identifies previous works using $N$ task-dependent prompt tokens for each task. (c) shows that our proposed SpeechComposer uses a unified architecture and composes primary tasks to more tasks with only five uniform prompt tokens.
  • Figure 2: The overall architecture of SpeechComposer. In this picture, we use the task of voice conversion and speech enhancement as an example. It demonstrates how composite tasks can be accomplished through the composition of primary tasks and the use of prompt tokens. Speech enhancement can also be composed in a similar manner with different enrollment speech tokens.