Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fish Audio S2 Technical Report

Shijia Liao, Yuxuan Wang, Songting Liu, Yifan Cheng, Ruoyi Zhang, Tianyu Li, Shidong Li, Yisheng Zheng, Xingwei Liu, Qingzheng Wang, Zhizhuo Zhou, Jiahua Liu, Xin Chen, Dawei Han

TL;DR

This work introduces Fish Audio S2, an open-sourced text-to-speech system featuring multi-speaker, multi-turn generation, and, most importantly, instruction-following control via natural-language descriptions, and releases model weights, fine-tuning code, and an SGLang-based inference engine.

Abstract

We introduce Fish Audio S2, an open-sourced text-to-speech system featuring multi-speaker, multi-turn generation, and, most importantly, instruction-following control via natural-language descriptions. To scale training, we develop a multi-stage training recipe together with a staged data pipeline covering video captioning and speech captioning, voice-quality assessment, and reward modeling. To push the frontier of open-source TTS, we release our model weights, fine-tuning code, and an SGLang-based inference engine. The inference engine is production-ready for streaming, achieving an RTF of 0.195 and a time-to-first-audio below 100 ms.Our code and weights are available on GitHub (https://github.com/fishaudio/fish-speech) and Hugging Face (https://huggingface.co/fishaudio/s2-pro). We highly encourage readers to visit https://fish.audio to try custom voices.

Fish Audio S2 Technical Report

TL;DR

This work introduces Fish Audio S2, an open-sourced text-to-speech system featuring multi-speaker, multi-turn generation, and, most importantly, instruction-following control via natural-language descriptions, and releases model weights, fine-tuning code, and an SGLang-based inference engine.

Abstract

We introduce Fish Audio S2, an open-sourced text-to-speech system featuring multi-speaker, multi-turn generation, and, most importantly, instruction-following control via natural-language descriptions. To scale training, we develop a multi-stage training recipe together with a staged data pipeline covering video captioning and speech captioning, voice-quality assessment, and reward modeling. To push the frontier of open-source TTS, we release our model weights, fine-tuning code, and an SGLang-based inference engine. The inference engine is production-ready for streaming, achieving an RTF of 0.195 and a time-to-first-audio below 100 ms.Our code and weights are available on GitHub (https://github.com/fishaudio/fish-speech) and Hugging Face (https://huggingface.co/fishaudio/s2-pro). We highly encourage readers to visit https://fish.audio to try custom voices.
Paper Structure (28 sections, 9 equations, 5 figures, 7 tables)

This paper contains 28 sections, 9 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Architecture
  3. Audio Tokenizer
  4. Dual-Autoregressive Generation
  5. Data Pipeline
  6. Speech Quality Model
  7. Rich-Transcription ASR Model
  8. Training
  9. Audio Tokenizer Training
  10. Pre-training and SFT
  11. RL-Based Post-Training
  12. Inference Engine
  13. Evaluation
  14. Objective Evaluation
  15. Seed-TTS-Eval
  16. ...and 13 more sections

Figures (5)

  • Figure 1: Fish Audio S2 is a multilingual, controllable, and expressive TTS system supporting long-form, multi-speaker, multi-turn generation with ultra-low TTFA and RTF.
  • Figure 2: Fish Audio S2 architecture.
  • Figure 3: Fish Audio S2 data pipeline.
  • Figure 4: Fish Audio S2 supports multi-speaker generation with fine-grained natural language control over prosody, emotion, and speaking style.
  • Figure 5: Training reward curves during RL-based post-training.