AnyEnhance: A Unified Generative Model with Prompt-Guidance and Self-Critic for Voice Enhancement

TL;DR

AnyEnhance addresses versatile voice enhancement across both speech and singing by employing a unified masked generative framework. It introduces prompt-guided in-context learning to incorporate a reference timbre for target speaker extraction and a self-critic sampling mechanism to iteratively refine outputs. The model operates with a two-stage semantic-acoustic architecture and a Descript Audio Codec tokenizer, augmented by a REPA alignment module, enabling simultaneous handling of denoising, dereverberation, declipping, super-resolution, and TSE without fine-tuning. Across diverse datasets and tasks, AnyEnhance achieves superior objective metrics and subjective quality, demonstrating strong generalization and practical potential for real-world voice processing in multi-domain settings.

Abstract

We introduce AnyEnhance, a unified generative model for voice enhancement that processes both speech and singing voices. Based on a masked generative model, AnyEnhance is capable of handling both speech and singing voices, supporting a wide range of enhancement tasks including denoising, dereverberation, declipping, super-resolution, and target speaker extraction, all simultaneously and without fine-tuning. AnyEnhance introduces a prompt-guidance mechanism for in-context learning, which allows the model to natively accept a reference speaker's timbre. In this way, it could boost enhancement performance when a reference audio is available and enable the target speaker extraction task without altering the underlying architecture. Moreover, we also introduce a self-critic mechanism into the generative process for masked generative models, yielding higher-quality outputs through iterative self-assessment and refinement. Extensive experiments on various enhancement tasks demonstrate AnyEnhance outperforms existing methods in terms of both objective metrics and subjective listening tests. Demo audios are publicly available at https://amphionspace.github.io/anyenhance. An open-source implementation is provided at https://github.com/viewfinder-annn/anyenhance-v1-ccf-aatc.

Figures (6)

• Figure 1: Model Architecture for AnyEnhance. It operates in two stages: (1) the semantic enhancement stage, where the encoder extracts and align semantic features from distorted and clean audio, (2) the acoustic enhancement stage, where the decoder predicts masked tokens using semantic features and acoustic tokens. Part of the clean audio can be retained for prompt-guidance (See Figure \ref{['fig:prompt']} for details), and a critic head enables self-critic training and sampling.
• Figure 2: Simulation of real-world live vocal recordings, including vocal effects, venue acoustics, and ambient noise.
• Figure 3: Comparison of model training with and without prompts. The inclusion of prompts enables the model to naturally perform TSE also achieving improved performance across other enhancement tasks.
• Figure 4: Comparison between Self-Critic and non-Self-Critic across three tasks: Librivox GSR, CCMusic SR, and VCTK TSE. Using Self-Critic Sampling consistently leads to better performance across varying sampling timesteps and metrics (OVRL, SpeechBERTScore, and Speaker Similarity).
• Figure 5: Comparison of the combined and individual effects of Prompt-Guidance and Self-Critic Sampling across different inference steps. Both mechanisms contribute to performance improvements, with prompt-guidance generally offering a larger gain.