FlowSE-GRPO: Training Flow Matching Speech Enhancement via Online Reinforcement Learning
Haoxu Wang, Biao Tian, Yiheng Jiang, Zexu Pan, Shengkui Zhao, Bin Ma, Daren Chen, Xiangang Li
TL;DR
This work addresses the gap in post-training alignment for generative speech enhancement by integrating online Group Relative Policy Optimization (GRPO) with a flow-matching SE model, FlowSE-GRPO. By converting the deterministic denoising process into a stochastic differential equation (SDE), the approach enables on-policy RL optimization and efficient fine-tuning with a windowed sampling strategy. A multi-metric reward formulation combining DNSMOS, speaker similarity, and SpeechBERTScore mitigates reward hacking and yields balanced improvements across perceptual quality and downstream metrics. Empirical results on DNS2020 show FlowSE-GRPO improves overall quality and robustness (No Reverb, With Reverb, Real Recording) with far fewer training steps than offline methods, providing practical guidelines for RL-based post-training of generative audio models. The study demonstrates the practicality and effectiveness of online post-training for generative speech models and highlights the importance of multi-objective reward design in preventing metric-specific optimization blind spots.
Abstract
Generative speech enhancement offers a promising alternative to traditional discriminative methods by modeling the distribution of clean speech conditioned on noisy inputs. Post-training alignment via reinforcement learning (RL) effectively aligns generative models with human preferences and downstream metrics in domains such as natural language processing, but its use in speech enhancement remains limited, especially for online RL. Prior work explores offline methods like Direct Preference Optimization (DPO); online methods such as Group Relative Policy Optimization (GRPO) remain largely uninvestigated. In this paper, we present the first successful integration of online GRPO into a flow-matching speech enhancement framework, enabling efficient post-training alignment to perceptual and task-oriented metrics with few update steps. Unlike prior GRPO work on Large Language Models, we adapt the algorithm to the continuous, time-series nature of speech and to the dynamics of flow-matching generative models. We show that optimizing a single reward yields rapid metric gains but often induces reward hacking that degrades audio fidelity despite higher scores. To mitigate this, we propose a multi-metric reward optimization strategy that balances competing objectives, substantially reducing overfitting and improving overall performance. Our experiments validate online GRPO for speech enhancement and provide practical guidance for RL-based post-training of generative audio models.