Self-supervised restoration of singing voice degraded by pitch shifting using shallow diffusion

TL;DR

This work tackles artifact-free pitch shifting by reframing it as a restoration task. It introduces a shallow diffusion model operating in mel space, conditioned on frame-level features $f_0$, volume, and ContentVec, and trained with self-supervised degraded pairs produced via WORLD-based forward/backward pitch shifts. Inference uses a deterministic DDIM-style schedule with $K=100$ steps to refine artifacted mel spectrograms, which are then vocoded by an $F_0$-aware NSF-HiFiGAN to produce waveforms. On unseen singing data, the approach achieves state-of-the-art distributional and pairwise reconstruction metrics, showing robust pitch fidelity and reduced artifacts, making it suitable for artifact-resistant vocal transposition workflows.

Abstract

Pitch shifting has been an essential feature in singing voice production. However, conventional signal processing approaches exhibit well known trade offs such as formant shifts and robotic coloration that becomes more severe at larger transposition jumps. This paper targets high quality pitch shifting for singing by reframing it as a restoration problem: given an audio track that has been pitch shifted (and thus contaminated by artifacts), we recover a natural sounding performance while preserving its melody and timing. Specifically, we use a lightweight, mel space diffusion model driven by frame level acoustic features such as f0, volume, and content features. We construct training pairs in a self supervised manner by applying pitch shifts and reversing them to simulate realistic artifacts while retaining ground truth. On a curated singing set, the proposed approach substantially reduces pitch shift artifacts compared to representative classical baselines, as measured by both statistical metrics and pairwise acoustic measures. The results suggest that restoration based pitch shifting could be a viable approach towards artifact resistant transposition in vocal production workflows.

Figures (1)

• Figure 1: Overview. Training: Original audio goes through a forward and backward pitch shift process via the WORLD vocoder. The shallow diffusion then denoises the artifact audio to reconstruct the mel. Notice $f_0$ is extracted from artifact Mel during training for higher robustness. Inference: Given a shifted audio, the diffusion model reconstructs it into a clean Mel-spectrogram. The $f0$ is directly input to the diffusion model during inference.