Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unifying Speech Recognition, Synthesis and Conversion with Autoregressive Transformers

Runyuan Cai, Yu Lin, Yiming Wang, Chunlin Fu, Xiaodong Zeng

TL;DR

GPA introduces a unified autoregressive framework for general-purpose speech that jointly handles TTS, ASR, and VC within a shared discrete token space. By combining BiCodec and GLM tokenizers and training on diverse data, GPA enables instruction-driven task switching without architectural changes and supports streaming, edge-friendly deployment with a compact 0.3B-parameter variant. Empirical results show competitive accuracy across benchmarks and strong throughput/latency characteristics, validating the practicality of a single autoregressive model for multiple speech tasks. This work reduces pipeline fragmentation and demonstrates the potential of token-based, end-to-end speech modeling as a scalable, deployable audio foundation. Overall, GPA highlights a viable path toward flexible, multi-task speech systems built on autoregressive generation and shared representations.

Abstract

Traditional speech systems typically rely on separate, task-specific models for text-to-speech (TTS), automatic speech recognition (ASR), and voice conversion (VC), resulting in fragmented pipelines that limit scalability, efficiency, and cross-task generalization. In this paper, we present General-Purpose Audio (GPA), a unified audio foundation model that integrates multiple core speech tasks within a single large language model (LLM) architecture. GPA operates on a shared discrete audio token space and supports instruction-driven task induction, enabling a single autoregressive model to flexibly perform TTS, ASR, and VC without architectural modifications. This unified design combines a fully autoregressive formulation over discrete speech tokens, joint multi-task training across speech domains, and a scalable inference pipeline that achieves high concurrency and throughput. The resulting model family supports efficient multi-scale deployment, including a lightweight 0.3B-parameter variant optimized for edge and resource-constrained environments. Together, these design choices demonstrate that a unified autoregressive architecture can achieve competitive performance across diverse speech tasks while remaining viable for low-latency, practical deployment.

Unifying Speech Recognition, Synthesis and Conversion with Autoregressive Transformers

TL;DR

GPA introduces a unified autoregressive framework for general-purpose speech that jointly handles TTS, ASR, and VC within a shared discrete token space. By combining BiCodec and GLM tokenizers and training on diverse data, GPA enables instruction-driven task switching without architectural changes and supports streaming, edge-friendly deployment with a compact 0.3B-parameter variant. Empirical results show competitive accuracy across benchmarks and strong throughput/latency characteristics, validating the practicality of a single autoregressive model for multiple speech tasks. This work reduces pipeline fragmentation and demonstrates the potential of token-based, end-to-end speech modeling as a scalable, deployable audio foundation. Overall, GPA highlights a viable path toward flexible, multi-task speech systems built on autoregressive generation and shared representations.

Abstract

Traditional speech systems typically rely on separate, task-specific models for text-to-speech (TTS), automatic speech recognition (ASR), and voice conversion (VC), resulting in fragmented pipelines that limit scalability, efficiency, and cross-task generalization. In this paper, we present General-Purpose Audio (GPA), a unified audio foundation model that integrates multiple core speech tasks within a single large language model (LLM) architecture. GPA operates on a shared discrete audio token space and supports instruction-driven task induction, enabling a single autoregressive model to flexibly perform TTS, ASR, and VC without architectural modifications. This unified design combines a fully autoregressive formulation over discrete speech tokens, joint multi-task training across speech domains, and a scalable inference pipeline that achieves high concurrency and throughput. The resulting model family supports efficient multi-scale deployment, including a lightweight 0.3B-parameter variant optimized for edge and resource-constrained environments. Together, these design choices demonstrate that a unified autoregressive architecture can achieve competitive performance across diverse speech tasks while remaining viable for low-latency, practical deployment.
Paper Structure (17 sections, 2 figures, 4 tables)

This paper contains 17 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. General Purpose Audio
  3. Motivation for a Fully Autoregressive Architecture
  4. Tokenization and Codec Design
  5. Joint Multi-Task Training
  6. Empirical Evaluation
  7. Training details
  8. From-Scratch Training.
  9. Pretraining Data.
  10. Supervised Fine-Tuning.
  11. Data Construction and Representation.
  12. Model Performance
  13. Inference
  14. latency and throughput
  15. Model Evaluations
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Architecture of the proposed GPA framework. The model utilizes a shared LLM backbone to unify three core audio tasks: understanding, generation, and editing. Depending on the task, the model processes different combinations of inputs via Semantic and Acoustic modules to generate the corresponding text or audio output.
  • Figure 2: Tokenization and task flow in GPA. Different audio tasks are realized by varying the input and output token compositions, all processed through a shared autoregressive backbone.