Aliasing-Free Neural Audio Synthesis

TL;DR

This work tackles fidelity limits in upsampling-based neural audio synthesis caused by aliasing from nonlinear activations and ConvTranspose upsampling. It introduces two anti-aliased module families: ADAA SnakeBeta activation with oversampling and a resampling-based upsampling path with a high-frequency deterministic prior, replacing traditional ConvTranspose to suppress aliased content. Building on these modules, the authors present Pupu-Vocoder and Pupu-Codec, along with extensive multi-domain experiments (speech, singing, music, and general audio) and a test-signal benchmark, showing improved subjective and objective quality—especially in singing and music—while maintaining competitive performance in speech. The methods are demonstrated to be domain-agnostic and efficient, suggesting broader applicability to aliasing-prone generative tasks beyond audio, such as in TF-domain representations for images.

Abstract

Neural vocoders and codecs reconstruct waveforms from acoustic representations, which directly impact the audio quality. Among existing methods, upsampling-based time-domain models are superior in both inference speed and synthesis quality, achieving state-of-the-art performance. Still, despite their success in producing perceptually natural sound, their synthesis fidelity remains limited due to the aliasing artifacts brought by the inadequately designed model architectures. In particular, the unconstrained nonlinear activation generates an infinite number of harmonics that exceed the Nyquist frequency, resulting in ``folded-back'' aliasing artifacts. The widely used upsampling layer, ConvTranspose, copies the mirrored low-frequency parts to fill the empty high-frequency region, resulting in ``mirrored'' aliasing artifacts. Meanwhile, the combination of its inherent periodicity and the mirrored DC bias also brings ``tonal artifact,'' resulting in constant-frequency ringing. This paper aims to solve these issues from a signal processing perspective. Specifically, we apply oversampling and anti-derivative anti-aliasing to the activation function to obtain its anti-aliased form, and replace the problematic ConvTranspose layer with resampling to avoid the ``tonal artifact'' and eliminate aliased components. Based on our proposed anti-aliased modules, we introduce Pupu-Vocoder and Pupu-Codec, and release high-quality pre-trained checkpoints to facilitate audio generation research. We build a test signal benchmark to illustrate the effectiveness of the anti-aliased modules, and conduct experiments on speech, singing voice, music, and audio to validate our proposed models. Experimental results confirm that our lightweight Pupu-Vocoder and Pupu-Codec models can easily outperform existing systems on singing voice, music, and audio, while achieving comparable performance on speech.

Figures (6)

• Figure 1: Illustration of different aliasing artifacts brought by the activation functions and upsampling layers. $F_S$ denotes the sampling rate, $F_N$ denotes the Nyquist frequency. The colored contours represent "folded-back" aliasing artifacts (orange), "mirrored" aliasing artifacts (green), DC bias (pink), and "tonal artifact" (purple), respectively.
• Figure 2: The main idea of our proposed anti-aliased activation function and upsampling layer. "$\rightarrow$" means "replacing the original inadequately designed model architecture with our proposed artifacts-free modules". $x_0$ is the latent representation obtained from the first Conv1D layer in the decoder. We employ a resampling layer (zero-interlacing + low-pass filtering) with a noise-like, high-pass filtered deterministic prior, obtained from the zero-interlaced $x_0$, to replace the problematic ConvTranspose layer. Additionally, we utilize an oversampled ADAA activation function to replace the original unconstrained one.
• Figure 3: Illustration of the equivalent filter frequency responses of linear and nearest interpolations. The orange contour and green dashed line represent the actual and ideal frequency responses, respectively. The red hatched area indicates the attenuation of the valid frequency region we want to retain, while the blue hatched area highlights the residual "mirrored" aliasing artifacts that the interpolation layer fails to remove.
• Figure 4: Architecture and training schemes of the proposed models. The Pupu-Codec consists of an encoder, a residual vector quantizer (RVQ) module, a decoder, and four different discriminators. "AF Conv Blocks" are obtained by modifying the convolution blocks used in BigVGAN bigvgan and DAC dac with our proposed anti-aliased activation and upsampling modules. Replacing the waveform input, encoder, and RVQ module with a mel-spectrogram as the input gives the Pupu-Vocoder model.
• Figure 5: Spectrogram visualization with a zoomed-in view of high-frequency harmonic components (around 16 kHz) regarding a representative singing voice example copy-synthesized by different neural vocoder and codec models.