Cross-domain Neural Pitch and Periodicity Estimation

TL;DR

This work tackles cross-domain neural pitch and periodicity estimation for speech and music, addressing limited generalization and speed in existing neural estimators. It introduces a comprehensive set of improvements to baseline predictors, plus a novel entropy-based periodicity extraction, and releases the Penn library for training, evaluation, and inference. The proposed FCNF0++ with entropy-based periodicity and unpitched-frame training achieves state-of-the-art results, with pitch accuracy and voicing decisions robust across domains and CPU speeds approaching those of DSP methods, while GPU inference is significantly faster. Together, these contributions enable fast, accurate, and generalizable pitch and periodicity estimation suitable for real-time and cross-domain audio applications.

Abstract

Pitch is a foundational aspect of our perception of audio signals. Pitch contours are commonly used to analyze speech and music signals and as input features for many audio tasks, including music transcription, singing voice synthesis, and prosody editing. In this paper, we describe a set of techniques for improving the accuracy of widely-used neural pitch and periodicity estimators to achieve state-of-the-art performance on both speech and music. We also introduce a novel entropy-based method for extracting periodicity and per-frame voiced-unvoiced classifications from statistical inference-based pitch estimators (e.g., neural networks), and show how to train a neural pitch estimator to simultaneously handle both speech and music data (i.e., cross-domain estimation) without performance degradation. Our estimator implementations run 11.2x faster than real-time on a Intel i9-9820X 10-core 3.30 GHz CPU$\unicode{x2014}$approaching the speed of state-of-the-art DSP-based pitch estimators$\unicode{x2014}$or 408x faster than real-time on a NVIDIA GeForce RTX 3090 GPU. We release all of our code and models as Pitch-Estimating Neural Networks (penn), an open-source, pip-installable Python module for training, evaluating, and performing inference with pitch- and periodicity-estimating neural networks. The code for penn is available at https://github.com/interactiveaudiolab/penn.

Figures (3)

• Figure 1: Pitch posteriorgrams produced by our reimplementation of the baseline FCNF0 pitch tracker ardaillon2019fully (middle) and our proposed FCNF0++ (bottom). The input audio (top) is the speech utterance "I am sitting in a room, different from the one you are in now". We apply softmax to each time frame to produce normalized distributions and take the natural log for better visualization. Greater brightness indicates higher probability. The y-axis frequency ranges are representative of the pitch bin ranges of the baseline and proposed models. Our proposed methods produce a sharper peak during pitched frames and encourage uniform probability in unpitched regions, making it easy to identify these regions algorithmically (Section \ref{['sec:periodicity']}).
• Figure 2: Hyperparameter landscape of the voiced/unvoiced threshold on the entropy-based periodicity estimate produced by FCNF0++ with (blue) and without (orange) our proposed unvoiced training strategy (Section \ref{['sec:new-methods']}) on PTDB and MDB-stem-synth. Stars indicate optimal F1 values found via a fine-grained binary search (Section \ref{['sec:periodicity-eval']}). Notice that---without unvoiced training strategy---the peak is narrow and can only be found via the fine-grained search (i.e., the orange star is not on the orange line). Our unvoiced training strategy of selecting a random bin (Section \ref{['sec:new-methods']}) improves the optimal F1 score of the model and produces state-of-the-art voiced/unvoiced classification F1 scores across a large region of the hyperparameter space.
• Figure 3: Cross-domain generalization across music and speech$|$ Test data distributions (blue), FCNF0++ model inference distributions (orange), and their overlap (gray) for the pitch bins of the test partitions of MDB-stem-synth (top) and PTDB (bottom). In other words, blue indicates where the model should have placed probability and did not, orange indicates where the model placed probability where it should not, and gray indicates the model is correct. The horizontal axis of each set of distributions is pitch bins from low (left) to high (right). Raw pitch accuracies (RPAs) with a 50 cent threshold (see Section \ref{['sec:pitch-eval']}) are provided below each set of distributions.