SwinSRGAN: Swin Transformer-based Generative Adversarial Network for High-Fidelity Speech Super-Resolution

TL;DR

SwinSRGAN introduces an end-to-end MDCT-domain speech super-resolution framework that uses a Swin Transformer U-Net to capture long-range spectro-temporal dependencies and a hybrid time-frequency discriminator focusing on the added high-band content. A sparse-aware, arcsinh-based loss alongside adversarial and feature-matching terms stabilizes training and preserves transients, enabling high-fidelity, vocoder-free upsampling up to 48 kHz in real time. Across VCTK and zero-shot HiFi-TTS evaluations, SwinSRGAN achieves superior LSD and ABX performance with competitive parameter efficiency, demonstrating strong generalization and practical impact for bandwidth extension. The work advances single-stage, non-diffusion-based speech SR with robust high-frequency reconstruction and efficient inference suitable for real-world deployment.

Abstract

Speech super-resolution (SR) reconstructs high-frequency content from low-resolution speech signals. Existing systems often suffer from representation mismatch in two-stage mel-vocoder pipelines and from over-smoothing of hallucinated high-band content by CNN-only generators. Diffusion and flow models are computationally expensive, and their robustness across domains and sampling rates remains limited. We propose SwinSRGAN, an end-to-end framework operating on Modified Discrete Cosine Transform (MDCT) magnitudes. It is a Swin Transformer-based U-Net that captures long-range spectro-temporal dependencies with a hybrid adversarial scheme combines time-domain MPD/MSD discriminators with a multi-band MDCT discriminator specialized for the high-frequency band. We employs a sparse-aware regularizer on arcsinh-compressed MDCT to better preserve transient components. The system upsamples inputs at various sampling rates to 48 kHz in a single pass and operates in real time. On standard benchmarks, SwinSRGAN reduces objective error and improves ABX preference scores. In zero-shot tests on HiFi-TTS without fine-tuning, it outperforms NVSR and mdctGAN, demonstrating strong generalization across datasets

Figures (3)

• Figure 1: Overall architecture of SwinSRGAN. The generator employs a U-Net architecture with a Swin Transformer backbone. It consists of multiple Encoder Stages for downsampling, a central RSTB as the bottleneck, and corresponding Decoder Stages for upsampling. Crucially, skip connections bridge the encoder and decoder paths to preserve high-fidelity spectral details. The discriminator adopts a hybrid architecture, combining time-domain discriminators (MPD/MSD) with a frequency-domain High-Band Multi-Band Discriminator (HB-MBD) to ensure both temporal coherence and spectral fidelity.
• Figure 2: Spectrogram comparison (16kHz$\rightarrow$48kHz). SwinSRGAN exhibits sharper harmonic structures, more complete high-band energy, and clearer transients with fewer artifacts, surpassing prior models in both detail and fidelity.
• Figure 3: ABX subjective test results of mdctGAN and SwinSRGAN on VCTK test set with 48 kHz target sampling rate.