Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Overview of the Amphion Toolkit (v0.2)

Jiaqi Li, Xueyao Zhang, Yuancheng Wang, Haorui He, Chaoren Wang, Li Wang, Huan Liao, Junyi Ao, Zeyu Xie, Yiqiao Huang, Junan Zhang, Zhizheng Wu

TL;DR

Amphion v0.2 introduces a scalable, open-source toolkit for audio, music, and speech generation, addressing entry barriers for junior researchers by pairing a 100K-hour multilingual dataset with a robust Emilia-Pipe data pipeline. The framework unifies state-of-the-art generative modalities, including neural audio codecs, autoregressive and masked speech language models, diffusion/flow-matching approaches, and advanced TTS/VC systems such as MaskGCT and Vevo, complemented by specialized datasets like SpMis and SD‑Eval. It also extends to practical tools for speech enhancement (AnyEnhance) and temporally controllable audio (PicoAudio), with extensive tutorials and open-source experiments demonstrating robust performance across multiple downstream tasks. The work highlights Amphion’s potential to accelerate research and real-world deployment in robust, controllable, and multilingual audio generation, while fostering an active ecosystem of models, datasets, and tutorials.

Abstract

Amphion is an open-source toolkit for Audio, Music, and Speech Generation, designed to lower the entry barrier for junior researchers and engineers in these fields. It provides a versatile framework that supports a variety of generation tasks and models. In this report, we introduce Amphion v0.2, the second major release developed in 2024. This release features a 100K-hour open-source multilingual dataset, a robust data preparation pipeline, and novel models for tasks such as text-to-speech, audio coding, and voice conversion. Furthermore, the report includes multiple tutorials that guide users through the functionalities and usage of the newly released models.

Overview of the Amphion Toolkit (v0.2)

TL;DR

Amphion v0.2 introduces a scalable, open-source toolkit for audio, music, and speech generation, addressing entry barriers for junior researchers by pairing a 100K-hour multilingual dataset with a robust Emilia-Pipe data pipeline. The framework unifies state-of-the-art generative modalities, including neural audio codecs, autoregressive and masked speech language models, diffusion/flow-matching approaches, and advanced TTS/VC systems such as MaskGCT and Vevo, complemented by specialized datasets like SpMis and SD‑Eval. It also extends to practical tools for speech enhancement (AnyEnhance) and temporally controllable audio (PicoAudio), with extensive tutorials and open-source experiments demonstrating robust performance across multiple downstream tasks. The work highlights Amphion’s potential to accelerate research and real-world deployment in robust, controllable, and multilingual audio generation, while fostering an active ecosystem of models, datasets, and tutorials.

Abstract

Amphion is an open-source toolkit for Audio, Music, and Speech Generation, designed to lower the entry barrier for junior researchers and engineers in these fields. It provides a versatile framework that supports a variety of generation tasks and models. In this report, we introduce Amphion v0.2, the second major release developed in 2024. This release features a 100K-hour open-source multilingual dataset, a robust data preparation pipeline, and novel models for tasks such as text-to-speech, audio coding, and voice conversion. Furthermore, the report includes multiple tutorials that guide users through the functionalities and usage of the newly released models.
Paper Structure (79 sections, 25 equations, 17 figures, 10 tables)

This paper contains 79 sections, 25 equations, 17 figures, 10 tables.

Table of Contents

  1. Introduction to the Amphion Toolkit
  2. What's new in Amphion v0.2
  3. The Fundamentals of Amphion
  4. Neural Audio Codecs
  5. Speech Language Models
  6. Autoregressive Language Models
  7. Masked Generative Models
  8. Diffusion and Flow-Matching Models
  9. Deepfake Detection
  10. Speech Generation Datasets
  11. Speech LLMs
  12. Technologies in Amphion
  13. Datasets
  14. Emilia Dataset
  15. SpMis Dataset
  16. ...and 64 more sections

Figures (17)

  • Figure 1: A High-level architecture of neural audio codecs.
  • Figure 2: An overview of the Emilia-Pipe preprocessing pipeline.
  • Figure 3: Duration statistics of the speech data by language.
  • Figure 4: Examples of spoken dialogues impacted by the rich information carried in speech (e.g. emotion, accent, age, environment)
  • Figure 5: An illustration of the proposed five-step dataset pipeline to create the Debatts-Data dataset. The pipeline consists of moderator detection, rebuttal session extraction, speaker diarization, overlap deletion and merging and speech enhancement and metadata extraction processes.
  • ...and 12 more figures