Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Real-Time Lyrics Alignment System Using Chroma And Phonetic Features For Classical Vocal Performance

Jiyun Park, Sangeon Yong, Taegyun Kwon, Juhan Nam

TL;DR

This work tackles real-time lyrics alignment for classical vocal performances by fusing chromagram (melodic) features with phonetic posteriorgrams (PPG) to capture both pitch and pronunciation. It introduces a two-phase pipeline: an offline phase that pre-aligns reference audio to lyrics using a symbolic MusicXML score and DTW-based alignment, and an online phase that performs live alignment with streaming audio via Online DTW and a 3-second window. The authors train a CRNN-based acoustic model to produce PPGs from a 66-bin mel-spectrogram and evaluate multiple feature combinations on a recast Winterreise dataset, winterreise_rt, achieving best results with Chroma + Phoneme5. The study demonstrates that combining melodic and phonetic features yields robust real-time alignment for classical singing and provides a public benchmark dataset for future research, with potential applications in live subtitling and concert experiences.

Abstract

The goal of real-time lyrics alignment is to take live singing audio as input and to pinpoint the exact position within given lyrics on the fly. The task can benefit real-world applications such as the automatic subtitling of live concerts or operas. However, designing a real-time model poses a great challenge due to the constraints of only using past input and operating within a minimal latency. Furthermore, due to the lack of datasets for real-time models for lyrics alignment, previous studies have mostly evaluated with private in-house datasets, resulting in a lack of standard evaluation methods. This paper presents a real-time lyrics alignment system for classical vocal performances with two contributions. First, we improve the lyrics alignment algorithm by finding an optimal combination of chromagram and phonetic posteriorgram (PPG) that capture melodic and phonetics features of the singing voice, respectively. Second, we recast the Schubert Winterreise Dataset (SWD) which contains multiple performance renditions of the same pieces as an evaluation set for the real-time lyrics alignment.

A Real-Time Lyrics Alignment System Using Chroma And Phonetic Features For Classical Vocal Performance

TL;DR

This work tackles real-time lyrics alignment for classical vocal performances by fusing chromagram (melodic) features with phonetic posteriorgrams (PPG) to capture both pitch and pronunciation. It introduces a two-phase pipeline: an offline phase that pre-aligns reference audio to lyrics using a symbolic MusicXML score and DTW-based alignment, and an online phase that performs live alignment with streaming audio via Online DTW and a 3-second window. The authors train a CRNN-based acoustic model to produce PPGs from a 66-bin mel-spectrogram and evaluate multiple feature combinations on a recast Winterreise dataset, winterreise_rt, achieving best results with Chroma + Phoneme5. The study demonstrates that combining melodic and phonetic features yields robust real-time alignment for classical singing and provides a public benchmark dataset for future research, with potential applications in live subtitling and concert experiences.

Abstract

The goal of real-time lyrics alignment is to take live singing audio as input and to pinpoint the exact position within given lyrics on the fly. The task can benefit real-world applications such as the automatic subtitling of live concerts or operas. However, designing a real-time model poses a great challenge due to the constraints of only using past input and operating within a minimal latency. Furthermore, due to the lack of datasets for real-time models for lyrics alignment, previous studies have mostly evaluated with private in-house datasets, resulting in a lack of standard evaluation methods. This paper presents a real-time lyrics alignment system for classical vocal performances with two contributions. First, we improve the lyrics alignment algorithm by finding an optimal combination of chromagram and phonetic posteriorgram (PPG) that capture melodic and phonetics features of the singing voice, respectively. Second, we recast the Schubert Winterreise Dataset (SWD) which contains multiple performance renditions of the same pieces as an evaluation set for the real-time lyrics alignment.
Paper Structure (11 sections, 4 figures, 1 table)

This paper contains 11 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. System Design
  3. Offline Phase
  4. Online Phase
  5. Acoustic Model
  6. Experiments
  7. Datasets
  8. Evaluation Metrics
  9. Feature Selection & Hyper-parameters
  10. Results & Discussions
  11. Conclusion

Figures (4)

  • Figure 1: An overview of the proposed real-time lyrics alignment system.
  • Figure 2: The pipeline of an offline phase aligning ref audio and lyrics by extracting score audio and annotation from the symbolic score.
  • Figure 3: The network architecture of the proposed acoustic model
  • Figure 4: Warping path results with and without phoneme features of 'No. 11, Frühlingstraum'. The red dots represent ground-truth pairs for voice notes, the purple dots with the warping path results of each model, and the white line with the silence part.