Lightweight Self-Supervised Detection of Fundamental Frequency and Accurate Probability of Voicing in Monophonic Music

TL;DR

This work tackles robust $F_0$ and voicing estimation for monophonic music under realistic artifacts and limited data. It introduces a lightweight self-supervised framework that enforces transposition-equivariant learning on a CQT front-end, paired with an EM-style iterative reweighting using Shift Cross-Entropy to down-weight unreliable frames and generate pseudo-labels for a small voicing classifier. The method trains exclusively on MedleyDB and demonstrates competitive cross-corpus performance and cross-instrument generalization, aided by a compact ResNeXt1D encoder and a Toeplitz pitch head. The approach yields a practical front-end for rapid instrument-specific conditioning in neural synthesis and DDSP-style pipelines, with strong potential for personalization and robust pitch tracking in real-world recordings.

Abstract

Reliable fundamental frequency (F 0) and voicing estimation is essential for neural synthesis, yet many pitch extractors depend on large labeled corpora and degrade under realistic recording artifacts. We propose a lightweight, fully self-supervised framework for joint F 0 estimation and voicing inference, designed for rapid single-instrument training from limited audio. Using transposition-equivariant learning on CQT features, we introduce an EM-style iterative reweighting scheme that uses Shift Cross-Entropy (SCE) consistency as a reliability signal to suppress uninformative noisy/unvoiced frames. The resulting weights provide confidence scores that enable pseudo-labeling for a separate lightweight voicing classifier without manual annotations. Trained on MedleyDB and evaluated on MDB-stem-synth ground truth, our method achieves competitive cross-corpus performance (RPA 95.84, RCA 96.24) and demonstrates cross-instrument generalization.

Figures (6)

• Figure 1: Model architecture. ResNeXt1D encoder over CQT frames followed by a Toeplitz linear classifier (implemented as a 1D convolution) producing pitch logits.
• Figure 2: Acoustic comparison: MedleyDB vs. MDB-stem-synth. (Top) MedleyDB recordings show broader partials, modulation sidebands, and exponential decay tails from room acoustics. (Bottom) MDB-stem-synth exhibits narrow harmonic tracks with abrupt terminations and exact silence (black regions). Real recordings retain residual energy from bleed and reverb where synthetic stems are perfectly zero.
• Figure 3: Learned weight behavior on MedleyDB. (Top) CQT spectrogram with predicted $F_0$ (black). (Middle) EM sample weights $w$ (green) remain high on sustained harmonics and drop during releases and transients. (Bottom) Derived voicing $\hat{v}$ (red) effectively gates voiced segments, automatically distinguishing stable pitch from ambiguous content.
• Figure 4: Histogram of EM sample weights on MedleyDB (log-count scale). Clear bimodality: most frames are confidently kept ($w\!\approx\!1$) or down-weighted ($w\!\approx\!0$).
• Figure 5: (a) Target audio (CQT). Clear stair-step contours from MedleyDB clarinet with narrow partials and clean inter-note gaps. (b) Synthesized audio (CQT) from $\hat{F}_{0}$ and $\hat{v}$. Harmonic ladders align with the target. The black gaps indicate voicing-controlled muting during predicted unvoiced spans. Low-frequency wedges at the very left/right arise from segment fade-in/out. (c) MDB-stem-synth Synthesis (CQT). Resynthesized audio exhibits narrow, high-contrast harmonic partials with hard on/offsets, exact silences (black gaps), and minimal modulation sidebands compared to real recordings.