Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Smule Renaissance Small: Efficient General-Purpose Vocal Restoration

Yongyi Zang, Chris Manchester, David Young, Ivan Ivanov, Jeffrey Lufkin, Martin Vladimirov, PJ Solomon, Svetoslav Kepchelev, Fei Yueh Chen, Dongting Cai, Teodor Naydenov, Randal Leistikow

TL;DR

Smule Renaissance Small (SRS) targets robust vocal restoration under compound real-world degradations directly in the complex-STFT domain. It introduces a compact bandwise generator with cross-band attention and per-band synthesis heads, augmented by phase-aware losses and a degradation-robust training pipeline to enable large analysis windows while remaining efficient for on-device inference. Empirically, SRS outperforms strong GAN baselines on the DNS 5 Challenge blind set and closely matches a costly flow-based system, and on the Extreme Degradation Bench it surpasses open-source baselines for singing and remains competitive for speech, despite no speech-specific training. The work also provides a realistic benchmarking resource (EDB) for severe multi-degradation scenarios and demonstrates practical real-time performance on consumer devices, with code and data released under the MIT License to support reproducible research.

Abstract

Vocal recordings on consumer devices commonly suffer from multiple concurrent degradations: noise, reverberation, band-limiting, and clipping. We present Smule Renaissance Small (SRS), a compact single-stage model that performs end-to-end vocal restoration directly in the complex STFT domain. By incorporating phase-aware losses, SRS enables large analysis windows for improved frequency resolution while achieving 10.5x real-time inference on iPhone 12 CPU at 48 kHz. On the DNS 5 Challenge blind set, despite no speech training, SRS outperforms a strong GAN baseline and closely matches a computationally expensive flow-matching system. To enable evaluation under realistic multi-degradation scenarios, we introduce the Extreme Degradation Bench (EDB): 87 singing and speech recordings captured under severe acoustic conditions. On EDB, SRS surpasses all open-source baselines on singing and matches commercial systems, while remaining competitive on speech despite no speech-specific training. We release both SRS and EDB under the MIT License.

Smule Renaissance Small: Efficient General-Purpose Vocal Restoration

TL;DR

Smule Renaissance Small (SRS) targets robust vocal restoration under compound real-world degradations directly in the complex-STFT domain. It introduces a compact bandwise generator with cross-band attention and per-band synthesis heads, augmented by phase-aware losses and a degradation-robust training pipeline to enable large analysis windows while remaining efficient for on-device inference. Empirically, SRS outperforms strong GAN baselines on the DNS 5 Challenge blind set and closely matches a costly flow-based system, and on the Extreme Degradation Bench it surpasses open-source baselines for singing and remains competitive for speech, despite no speech-specific training. The work also provides a realistic benchmarking resource (EDB) for severe multi-degradation scenarios and demonstrates practical real-time performance on consumer devices, with code and data released under the MIT License to support reproducible research.

Abstract

Vocal recordings on consumer devices commonly suffer from multiple concurrent degradations: noise, reverberation, band-limiting, and clipping. We present Smule Renaissance Small (SRS), a compact single-stage model that performs end-to-end vocal restoration directly in the complex STFT domain. By incorporating phase-aware losses, SRS enables large analysis windows for improved frequency resolution while achieving 10.5x real-time inference on iPhone 12 CPU at 48 kHz. On the DNS 5 Challenge blind set, despite no speech training, SRS outperforms a strong GAN baseline and closely matches a computationally expensive flow-matching system. To enable evaluation under realistic multi-degradation scenarios, we introduce the Extreme Degradation Bench (EDB): 87 singing and speech recordings captured under severe acoustic conditions. On EDB, SRS surpasses all open-source baselines on singing and matches commercial systems, while remaining competitive on speech despite no speech-specific training. We release both SRS and EDB under the MIT License.
Paper Structure (24 sections, 6 equations, 2 figures, 3 tables)

This paper contains 24 sections, 6 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Model Architecture
  4. Input representation.
  5. Bandwise decomposition.
  6. Per-band feature extraction.
  7. Band–Sequence block (repeated $L$ times).
  8. Per-band synthesis and spectral reassembly.
  9. Computational complexity and design rationale.
  10. Training Setup
  11. Input–output notation.
  12. Reconstruction objective.
  13. Adversarial objectives.
  14. Optimization and regularization.
  15. Data
  16. ...and 9 more sections

Figures (2)

  • Figure 1: Overall results for all systems on Extreme Degradation Bench.
  • Figure 2: Pairwise win/tie/lose results for all systems on Extreme Degradation Bench.