Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VietLyrics: A Large-Scale Dataset and Models for Vietnamese Automatic Lyrics Transcription

Quoc Anh Nguyen, Bernard Cheng, Kelvin Soh

TL;DR

The paper introduces VietLyrics, the first large-scale Vietnamese ALT dataset (647 hours) with line-level lyrics and metadata, and demonstrates that fine-tuning Whisper models yields substantial gains over existing multilingual ALT systems. It details data collection from zingmp3.vn, preprocessing to 16 kHz, and 30-second audio-lyrics segmentation, followed by fine-tuning Whisper across model sizes. The best model, Whisper-large-v2, achieves WER 24.61% and CER 17.14% (case-sensitive) and 20.52% WER under lowercase, validating effectiveness across dialects and genres. The authors release VietLyrics and three Whisper checkpoints, advancing Vietnamese music computing and ALT research in low-resource languages.

Abstract

Automatic Lyrics Transcription (ALT) for Vietnamese music presents unique challenges due to its tonal complexity and dialectal variations, but remains largely unexplored due to the lack of a dedicated dataset. Therefore, we curated the first large-scale Vietnamese ALT dataset (VietLyrics), comprising 647 hours of songs with line-level aligned lyrics and metadata to address these issues. Our evaluation of current ASRbased approaches reveal significant limitations, including frequent transcription errors and hallucinations in non-vocal segments. To improve performance, we fine-tuned Whisper models on the VietLyrics dataset, achieving superior results compared to existing multilingual ALT systems, including LyricWhiz. We publicly release VietLyrics and our models, aiming to advance Vietnamese music computing research while demonstrating the potential of this approach for ALT in low-resource language and music.

VietLyrics: A Large-Scale Dataset and Models for Vietnamese Automatic Lyrics Transcription

TL;DR

The paper introduces VietLyrics, the first large-scale Vietnamese ALT dataset (647 hours) with line-level lyrics and metadata, and demonstrates that fine-tuning Whisper models yields substantial gains over existing multilingual ALT systems. It details data collection from zingmp3.vn, preprocessing to 16 kHz, and 30-second audio-lyrics segmentation, followed by fine-tuning Whisper across model sizes. The best model, Whisper-large-v2, achieves WER 24.61% and CER 17.14% (case-sensitive) and 20.52% WER under lowercase, validating effectiveness across dialects and genres. The authors release VietLyrics and three Whisper checkpoints, advancing Vietnamese music computing and ALT research in low-resource languages.

Abstract

Automatic Lyrics Transcription (ALT) for Vietnamese music presents unique challenges due to its tonal complexity and dialectal variations, but remains largely unexplored due to the lack of a dedicated dataset. Therefore, we curated the first large-scale Vietnamese ALT dataset (VietLyrics), comprising 647 hours of songs with line-level aligned lyrics and metadata to address these issues. Our evaluation of current ASRbased approaches reveal significant limitations, including frequent transcription errors and hallucinations in non-vocal segments. To improve performance, we fine-tuned Whisper models on the VietLyrics dataset, achieving superior results compared to existing multilingual ALT systems, including LyricWhiz. We publicly release VietLyrics and our models, aiming to advance Vietnamese music computing research while demonstrating the potential of this approach for ALT in low-resource language and music.
Paper Structure (16 sections, 3 figures, 5 tables)

This paper contains 16 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Dataset Scarcity
  4. Limitations in Existing Solutions
  5. Proposed Method
  6. Dataset
  7. Data Collection & Processing
  8. Data Analysis
  9. Modeling
  10. Data Pre-processing
  11. Fine-Tuning
  12. Experiments and Results
  13. Finetuned Models
  14. <nospeech> suppression
  15. Conclusion
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Distribution of songs by genres. Extracted songs covers a wide range of genres, however a lot is missing.
  • Figure 2: Distribution of the dataset based on predicted genders and regional dialects.
  • Figure 3: Our Fine-tuned Whisper-large performance WER (%), evaluated based on predicted genders and regional dialects.