Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matching

Yuepeng Jiang, Huakang Chen, Ziqian Ning, Jixun Yao, Zerui Han, Di Wu, Meng Meng, Jian Luan, Zhonghua Fu, Lei Xie

TL;DR

DiffRhythm 2 tackles long-form song generation by coupling lyric-vocal alignment with high-fidelity accompaniment under a semi-autoregressive framework. It introduces block flow matching, a stochastic block REPA loss, and cross-pair preference optimization to balance multiple human preferences without model merging, enabled by a low-frame-rate ($5$ Hz) music VAE for tractable long sequences. The method achieves improved lyric alignment and overall musicality, outperforming open-source baselines and approaching commercial systems, while enabling efficient inference through block-level caching. These advancements advance practical, controllable song generation and raise considerations for responsible deployment and data use in RLHF contexts.

Abstract

Generating full-length, high-quality songs is challenging, as it requires maintaining long-term coherence both across text and music modalities and within the music modality itself. Existing non-autoregressive (NAR) frameworks, while capable of producing high-quality songs, often struggle with the alignment between lyrics and vocal. Concurrently, catering to diverse musical preferences necessitates reinforcement learning from human feedback (RLHF). However, existing methods often rely on merging multiple models during multi-preference optimization, which results in significant performance degradation. To address these challenges, we introduce DiffRhythm 2, an end-to-end framework designed for high-fidelity, controllable song generation. To tackle the lyric alignment problem, DiffRhythm 2 employs a semi-autoregressive architecture based on block flow matching. This design enables faithful alignment of lyrics to singing vocals without relying on external labels and constraints, all while preserving the high generation quality and efficiency of NAR models. To make this framework computationally tractable for long sequences, we implement a music variational autoencoder (VAE) that achieves a low frame rate of 5 Hz while still enabling high-fidelity audio reconstruction. In addition, to overcome the limitations of multi-preference optimization in RLHF, we propose cross-pair preference optimization. This method effectively mitigates the performance drop typically associated with model merging, allowing for more robust optimization across diverse human preferences. We further enhance musicality and structural coherence by introducing stochastic block representation alignment loss.

DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matching

TL;DR

DiffRhythm 2 tackles long-form song generation by coupling lyric-vocal alignment with high-fidelity accompaniment under a semi-autoregressive framework. It introduces block flow matching, a stochastic block REPA loss, and cross-pair preference optimization to balance multiple human preferences without model merging, enabled by a low-frame-rate ( Hz) music VAE for tractable long sequences. The method achieves improved lyric alignment and overall musicality, outperforming open-source baselines and approaching commercial systems, while enabling efficient inference through block-level caching. These advancements advance practical, controllable song generation and raise considerations for responsible deployment and data use in RLHF contexts.

Abstract

Generating full-length, high-quality songs is challenging, as it requires maintaining long-term coherence both across text and music modalities and within the music modality itself. Existing non-autoregressive (NAR) frameworks, while capable of producing high-quality songs, often struggle with the alignment between lyrics and vocal. Concurrently, catering to diverse musical preferences necessitates reinforcement learning from human feedback (RLHF). However, existing methods often rely on merging multiple models during multi-preference optimization, which results in significant performance degradation. To address these challenges, we introduce DiffRhythm 2, an end-to-end framework designed for high-fidelity, controllable song generation. To tackle the lyric alignment problem, DiffRhythm 2 employs a semi-autoregressive architecture based on block flow matching. This design enables faithful alignment of lyrics to singing vocals without relying on external labels and constraints, all while preserving the high generation quality and efficiency of NAR models. To make this framework computationally tractable for long sequences, we implement a music variational autoencoder (VAE) that achieves a low frame rate of 5 Hz while still enabling high-fidelity audio reconstruction. In addition, to overcome the limitations of multi-preference optimization in RLHF, we propose cross-pair preference optimization. This method effectively mitigates the performance drop typically associated with model merging, allowing for more robust optimization across diverse human preferences. We further enhance musicality and structural coherence by introducing stochastic block representation alignment loss.
Paper Structure (26 sections, 6 equations, 3 figures, 8 tables, 2 algorithms)

This paper contains 26 sections, 6 equations, 3 figures, 8 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Flow Matching and Autoregressive Diffusion Model
  4. Song Generation
  5. Method
  6. Overview
  7. Music VAE
  8. Block Flow Matching
  9. Definition
  10. Details of Application
  11. Variable Length Generation
  12. Training and Inference Procedures
  13. Stochastic Block REPA Loss
  14. Cross-Pair Preference Optimization
  15. Experimental Setup
  16. ...and 11 more sections

Figures (3)

  • Figure 1: Overview architecture of our proposed DiffRhythm 2. Either text description or audio can specify the style prompt.
  • Figure 2: The right panel shows the block-level latent sequence structure of the inputs and outputs, while the left panel shows the corresponding attention mask applied during training. Note that the block size here is 2.
  • Figure : Block Flow Matching Training