A Mamba-based Network for Semi-supervised Singing Melody Extraction Using Confidence Binary Regularization

TL;DR

This work tackles singing melody extraction (SME) under limited labeled data and the inefficiencies of transformer-based approaches by introducing SpectMamba, a linear-time encoder derived from vision mamba concepts. The model combines a coarse-to-fine note-f0 decoder with a confidence-based semi-supervised learning signal (CBR) to leverage unlabeled data. Key contributions include (i) a fast, bidirectional state-space encoder, (ii) a note-f0 decoder that aligns note-level structure with f0 estimation, and (iii) a CBR module that enforces consistency between weak and strong augmentations. Empirically, SpectMamba yields higher OA and reduces octave errors across public SME datasets while offering significantly faster inference and lower memory usage than transformer baselines.

Abstract

Singing melody extraction (SME) is a key task in the field of music information retrieval. However, existing methods are facing several limitations: firstly, prior models use transformers to capture the contextual dependencies, which requires quadratic computation resulting in low efficiency in the inference stage. Secondly, prior works typically rely on frequencysupervised methods to estimate the fundamental frequency (f0), which ignores that the musical performance is actually based on notes. Thirdly, transformers typically require large amounts of labeled data to achieve optimal performances, but the SME task lacks of sufficient annotated data. To address these issues, in this paper, we propose a mamba-based network, called SpectMamba, for semi-supervised singing melody extraction using confidence binary regularization. In particular, we begin by introducing vision mamba to achieve computational linear complexity. Then, we propose a novel note-f0 decoder that allows the model to better mimic the musical performance. Further, to alleviate the scarcity of the labeled data, we introduce a confidence binary regularization (CBR) module to leverage the unlabeled data by maximizing the probability of the correct classes. The proposed method is evaluated on several public datasets and the conducted experiments demonstrate the effectiveness of our proposed method.

Figures (3)

• Figure 1: The framework of the proposed SpectMamba.
• Figure 2: Inference Time vs GPU Usage Memory in ADC2004.
• Figure 3: Visualization of singing melody extraction results on two opera songs using ${{\rm{S}}^{\rm{2}}}{\rm{Former}}$ and the proposed SpectMamba.