Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VietASR: Achieving Industry-level Vietnamese ASR with 50-hour labeled data and Large-Scale Speech Pretraining

Jianheng Zhuo, Yifan Yang, Yiwen Shao, Yong Xu, Dong Yu, Kai Yu, Xie Chen

TL;DR

VietASR tackles the challenge of building high-quality ASR for low-resource Vietnamese by combining a large pool of unlabeled data with a small labeled set. It adapts HuBERT-style pre-training to a lightweight Zipformer backbone and introduces an ASR-biased supervised codebook to align pre-training with downstream ASR tasks, enabling four-stage iterative pre-training and fine-tuning. The approach achieves state-of-the-art results on Vietnamese benchmarks, outperforming Whisper Large-v3 and commercial systems with a compact 68M-parameter encoder. Its streaming variant also delivers strong performance, illustrating practical deployment potential and promising scalability to other low-resource languages through open-sourcing.

Abstract

Automatic speech recognition (ASR) has made remarkable progress but heavily relies on large-scale labeled data, which is scarce for low-resource languages like Vietnamese. While existing systems such as Whisper, USM, and MMS achieve promising performance, their efficacy remains inadequate in terms of training costs, latency, and accessibility. To address these issues, we propose VietASR, a novel ASR training pipeline that leverages vast amounts of unlabeled data and a small set of labeled data. Through multi-iteration ASR-biased self-supervised learning on a large-scale unlabeled dataset, VietASR offers a cost-effective and practical solution for enhancing ASR performance. Experiments demonstrate that pre-training on 70,000-hour unlabeled data and fine-tuning on merely 50-hour labeled data yield a lightweight but powerful ASR model. It outperforms Whisper Large-v3 and commercial ASR systems on real-world data. Our code and models will be open-sourced to facilitate research in low-resource ASR.

VietASR: Achieving Industry-level Vietnamese ASR with 50-hour labeled data and Large-Scale Speech Pretraining

TL;DR

VietASR tackles the challenge of building high-quality ASR for low-resource Vietnamese by combining a large pool of unlabeled data with a small labeled set. It adapts HuBERT-style pre-training to a lightweight Zipformer backbone and introduces an ASR-biased supervised codebook to align pre-training with downstream ASR tasks, enabling four-stage iterative pre-training and fine-tuning. The approach achieves state-of-the-art results on Vietnamese benchmarks, outperforming Whisper Large-v3 and commercial systems with a compact 68M-parameter encoder. Its streaming variant also delivers strong performance, illustrating practical deployment potential and promising scalability to other low-resource languages through open-sourcing.

Abstract

Automatic speech recognition (ASR) has made remarkable progress but heavily relies on large-scale labeled data, which is scarce for low-resource languages like Vietnamese. While existing systems such as Whisper, USM, and MMS achieve promising performance, their efficacy remains inadequate in terms of training costs, latency, and accessibility. To address these issues, we propose VietASR, a novel ASR training pipeline that leverages vast amounts of unlabeled data and a small set of labeled data. Through multi-iteration ASR-biased self-supervised learning on a large-scale unlabeled dataset, VietASR offers a cost-effective and practical solution for enhancing ASR performance. Experiments demonstrate that pre-training on 70,000-hour unlabeled data and fine-tuning on merely 50-hour labeled data yield a lightweight but powerful ASR model. It outperforms Whisper Large-v3 and commercial ASR systems on real-world data. Our code and models will be open-sourced to facilitate research in low-resource ASR.

Paper Structure

This paper contains 15 sections, 3 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. HuBERT and Its Variants
  3. VietASR
  4. HuBERT-style Pre-training with Zipformer
  5. Zipformer Backbone
  6. Mask Strategy
  7. Loss Function
  8. Speech Pre-training Pipeline with ASR-biased SSL
  9. Experiments
  10. Experimental Setups
  11. Comparison to Open-Source and Commercial Systems
  12. Analysis of the Impact of Supervised Codebook
  13. Performance of Streaming Systems
  14. Conclusion
  15. Acknowledgement

Figures (1)

  • Figure 1: An overview of the proposed VietASR. The training pipeline consists of four stages: (1) initial ASR training, (2) label extraction, (3) pre-training, and (4) fine-tuning. Components within the dashed box are reused in the next stage. In the first iteration, Stage 2 reuses the encoder trained in Stage 1 for label extraction; in subsequent iterations, Stage 2 reuses the encoder from Stage 4 of the previous iteration, enabling iterative refinement and continuous model improvement.