Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FireRedASR: Open-Source Industrial-Grade Mandarin Speech Recognition Models from Encoder-Decoder to LLM Integration

Kai-Tuo Xu, Feng-Long Xie, Xu Tang, Yao Hu

TL;DR

The paper addresses the need for open-source, industrial-grade Mandarin ASR by introducing FireRedASR-LLM and FireRedASR-AED, two architectures that balance accuracy and efficiency. FireRedASR-LLM integrates an Encoder-Adapter-LLM workflow to leverage LLM capabilities, achieving state-of-the-art-like CER on Mandarin benchmarks, while FireRedASR-AED delivers competitive results with a smaller footprint suitable for resource-constrained settings. Comprehensive evaluations demonstrate strong performance across public Mandarin benchmarks, diverse real-world scenarios, singing lyrics, and cross-lingual benchmarks, with notable generalization to dialects and English speech. The authors release pre-trained weights and inference code to foster open research and practical deployment, and discuss data quality, training strategies, and architectural choices as reasons for their success.

Abstract

We present FireRedASR, a family of large-scale automatic speech recognition (ASR) models for Mandarin, designed to meet diverse requirements in superior performance and optimal efficiency across various applications. FireRedASR comprises two variants: FireRedASR-LLM: Designed to achieve state-of-the-art (SOTA) performance and to enable seamless end-to-end speech interaction. It adopts an Encoder-Adapter-LLM framework leveraging large language model (LLM) capabilities. On public Mandarin benchmarks, FireRedASR-LLM (8.3B parameters) achieves an average Character Error Rate (CER) of 3.05%, surpassing the latest SOTA of 3.33% with an 8.4% relative CER reduction (CERR). It demonstrates superior generalization capability over industrial-grade baselines, achieving 24%-40% CERR in multi-source Mandarin ASR scenarios such as video, live, and intelligent assistant. FireRedASR-AED: Designed to balance high performance and computational efficiency and to serve as an effective speech representation module in LLM-based speech models. It utilizes an Attention-based Encoder-Decoder (AED) architecture. On public Mandarin benchmarks, FireRedASR-AED (1.1B parameters) achieves an average CER of 3.18%, slightly worse than FireRedASR-LLM but still outperforming the latest SOTA model with over 12B parameters. It offers a more compact size, making it suitable for resource-constrained applications. Moreover, both models exhibit competitive results on Chinese dialects and English speech benchmarks and excel in singing lyrics recognition. To advance research in speech processing, we release our models and inference code at https://github.com/FireRedTeam/FireRedASR.

FireRedASR: Open-Source Industrial-Grade Mandarin Speech Recognition Models from Encoder-Decoder to LLM Integration

TL;DR

The paper addresses the need for open-source, industrial-grade Mandarin ASR by introducing FireRedASR-LLM and FireRedASR-AED, two architectures that balance accuracy and efficiency. FireRedASR-LLM integrates an Encoder-Adapter-LLM workflow to leverage LLM capabilities, achieving state-of-the-art-like CER on Mandarin benchmarks, while FireRedASR-AED delivers competitive results with a smaller footprint suitable for resource-constrained settings. Comprehensive evaluations demonstrate strong performance across public Mandarin benchmarks, diverse real-world scenarios, singing lyrics, and cross-lingual benchmarks, with notable generalization to dialects and English speech. The authors release pre-trained weights and inference code to foster open research and practical deployment, and discuss data quality, training strategies, and architectural choices as reasons for their success.

Abstract

We present FireRedASR, a family of large-scale automatic speech recognition (ASR) models for Mandarin, designed to meet diverse requirements in superior performance and optimal efficiency across various applications. FireRedASR comprises two variants: FireRedASR-LLM: Designed to achieve state-of-the-art (SOTA) performance and to enable seamless end-to-end speech interaction. It adopts an Encoder-Adapter-LLM framework leveraging large language model (LLM) capabilities. On public Mandarin benchmarks, FireRedASR-LLM (8.3B parameters) achieves an average Character Error Rate (CER) of 3.05%, surpassing the latest SOTA of 3.33% with an 8.4% relative CER reduction (CERR). It demonstrates superior generalization capability over industrial-grade baselines, achieving 24%-40% CERR in multi-source Mandarin ASR scenarios such as video, live, and intelligent assistant. FireRedASR-AED: Designed to balance high performance and computational efficiency and to serve as an effective speech representation module in LLM-based speech models. It utilizes an Attention-based Encoder-Decoder (AED) architecture. On public Mandarin benchmarks, FireRedASR-AED (1.1B parameters) achieves an average CER of 3.18%, slightly worse than FireRedASR-LLM but still outperforming the latest SOTA model with over 12B parameters. It offers a more compact size, making it suitable for resource-constrained applications. Moreover, both models exhibit competitive results on Chinese dialects and English speech benchmarks and excel in singing lyrics recognition. To advance research in speech processing, we release our models and inference code at https://github.com/FireRedTeam/FireRedASR.
Paper Structure (10 sections, 1 figure, 5 tables)

This paper contains 10 sections, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. FireRedASR
  3. FireRedASR-AED: Attention-based Encoder-Decoder ASR model
  4. FireRedASR-LLM: Encoder-Adapter-LLM-based ASR model
  5. Evaluation
  6. Evaluation on Public Mandarin ASR Benchmarks
  7. Evaluation on Multi-source Mandarin Speech and Singing Benchmarks
  8. Evaluation on Public Chinese Dialect and English ASR Benchmarks
  9. Discussion
  10. Conclusion

Figures (1)

  • Figure 1: Architecture of FireRedASR-LLM (left), FireRedASR-AED (bottom right), and Adapter.