Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ACE-Step: A Step Towards Music Generation Foundation Model

Junmin Gong, Sean Zhao, Sen Wang, Shengyuan Xu, Joe Guo

TL;DR

<3-5 sentence high-level summary> ACE-Step addresses the open-source gap in long-form, lyric-aligned music generation by integrating diffusion-based synthesis with a Deep Compression AutoEncoder and a lightweight Linear Diffusion Transformer, enabling fast generation and high musical coherence. It introduces Representation Alignment (REPA) with pre-trained SSL models (MERT, mHuBERT) to strengthen lyric fidelity and semantic control, while supporting flexible conditioning from text, lyrics, and speaker identity. The work demonstrates strong human and automatic evaluations, competitive with commercial systems on open benchmarks, and enables extensive editing and downstream fine-tuning with LoRA and ControlNet-inspired conditioning. As a foundation-model-style platform, ACE-Step aims to catalyze open, collaborative development and practical music-creation workflows for artists and developers alike.

Abstract

We introduce ACE-Step, a novel open-source foundation model for music generation that overcomes key limitations of existing approaches and achieves state-of-the-art performance through a holistic architectural design. Current methods face inherent trade-offs between generation speed, musical coherence, and controllability. For example, LLM-based models (e.g. Yue, SongGen) excel at lyric alignment but suffer from slow inference and structural artifacts. Diffusion models (e.g. DiffRhythm), on the other hand, enable faster synthesis but often lack long-range structural coherence. ACE-Step bridges this gap by integrating diffusion-based generation with Sana's Deep Compression AutoEncoder (DCAE) and a lightweight linear transformer. It also leverages MERT and m-hubert to align semantic representations (REPA) during training, allowing rapid convergence. As a result, our model synthesizes up to 4 minutes of music in just 20 seconds on an A100 GPU-15x faster than LLM-based baselines-while achieving superior musical coherence and lyric alignment across melody, harmony, and rhythm metrics. Moreover, ACE-Step preserves fine-grained acoustic details, enabling advanced control mechanisms such as voice cloning, lyric editing, remixing, and track generation (e.g. lyric2vocal, singing2accompaniment). Rather than building yet another end-to-end text-to-music pipeline, our vision is to establish a foundation model for music AI: a fast, general-purpose, efficient yet flexible architecture that makes it easy to train subtasks on top of it. This paves the way for the development of powerful tools that seamlessly integrate into the creative workflows of music artists, producers, and content creators. In short, our goal is to build a stable diffusion moment for music. The code, the model weights and the demo are available at: https://ace-step.github.io/.

ACE-Step: A Step Towards Music Generation Foundation Model

TL;DR

<3-5 sentence high-level summary> ACE-Step addresses the open-source gap in long-form, lyric-aligned music generation by integrating diffusion-based synthesis with a Deep Compression AutoEncoder and a lightweight Linear Diffusion Transformer, enabling fast generation and high musical coherence. It introduces Representation Alignment (REPA) with pre-trained SSL models (MERT, mHuBERT) to strengthen lyric fidelity and semantic control, while supporting flexible conditioning from text, lyrics, and speaker identity. The work demonstrates strong human and automatic evaluations, competitive with commercial systems on open benchmarks, and enables extensive editing and downstream fine-tuning with LoRA and ControlNet-inspired conditioning. As a foundation-model-style platform, ACE-Step aims to catalyze open, collaborative development and practical music-creation workflows for artists and developers alike.

Abstract

We introduce ACE-Step, a novel open-source foundation model for music generation that overcomes key limitations of existing approaches and achieves state-of-the-art performance through a holistic architectural design. Current methods face inherent trade-offs between generation speed, musical coherence, and controllability. For example, LLM-based models (e.g. Yue, SongGen) excel at lyric alignment but suffer from slow inference and structural artifacts. Diffusion models (e.g. DiffRhythm), on the other hand, enable faster synthesis but often lack long-range structural coherence. ACE-Step bridges this gap by integrating diffusion-based generation with Sana's Deep Compression AutoEncoder (DCAE) and a lightweight linear transformer. It also leverages MERT and m-hubert to align semantic representations (REPA) during training, allowing rapid convergence. As a result, our model synthesizes up to 4 minutes of music in just 20 seconds on an A100 GPU-15x faster than LLM-based baselines-while achieving superior musical coherence and lyric alignment across melody, harmony, and rhythm metrics. Moreover, ACE-Step preserves fine-grained acoustic details, enabling advanced control mechanisms such as voice cloning, lyric editing, remixing, and track generation (e.g. lyric2vocal, singing2accompaniment). Rather than building yet another end-to-end text-to-music pipeline, our vision is to establish a foundation model for music AI: a fast, general-purpose, efficient yet flexible architecture that makes it easy to train subtasks on top of it. This paves the way for the development of powerful tools that seamlessly integrate into the creative workflows of music artists, producers, and content creators. In short, our goal is to build a stable diffusion moment for music. The code, the model weights and the demo are available at: https://ace-step.github.io/.

Paper Structure

This paper contains 44 sections, 3 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Generative Modeling for Audio and Music
  4. Open-Source Works
  5. Closed-Source Works and Our Motivation
  6. Core Generative Techniques and Components
  7. Method
  8. Data Foundation
  9. Dataset Curation and Quality Control
  10. Annotation and Conditioning Signals
  11. Input Strategies and Robustness
  12. Addressing Lyric-Audio Alignment Challenges
  13. Model Architecture
  14. Audio Autoencoder: Music-DCAE
  15. Core Diffusion Model: Linear DiT
  16. ...and 29 more sections

Figures (5)

  • Figure 1: ACE-Step Framework
  • Figure 2: Human evaluation results comparing ACE-Step against three baseline models (Hailuo, Suno v3, and Udio v1) across four dimensions. Scores are normalized to 0-100.
  • Figure 3: Comparison of (a) Genre Scores and (b) Lyric Scores Across Models.
  • Figure 4: Comparison of (a) Audiobox Aesthetics Metrics and (b) SongEval Musicality Metrics Across Models.
  • Figure 5: An overview of the controllable features and specialized downstream applications of our model.