JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation

TL;DR

JEN-1 Composer presents a unified, diffusion-based framework for high-fidelity multi-track music generation that models marginal, conditional, and joint distributions within a single audio latent diffusion model. By introducing multi-track input-output representations, per-track timesteps, and task-prefix prompts, combined with a progressive curriculum and a Human-AI co-composition workflow, the approach enables track-wise generation and iterative refinement aligned with human creativity. Empirical results show state-of-the-art performance in controllability and audio quality, with strong CLAP scores, favorable RPR, and competitive FAD on zero-shot multi-track benchmarks, underscoring the method's practical impact for professional music production. The work advances interactive AI-assisted music creation by bridging high-fidelity audio synthesis with flexible control and human-in-the-loop editing, backed by rigorous ablations and comprehensive evaluations.

Abstract

With rapid advances in generative artificial intelligence, the text-to-music synthesis task has emerged as a promising direction for music generation. Nevertheless, achieving precise control over multi-track generation remains an open challenge. While existing models excel in directly generating multi-track mix, their limitations become evident when it comes to composing individual tracks and integrating them in a controllable manner. This departure from the typical workflows of professional composers hinders the ability to refine details in specific tracks. To address this gap, we propose JEN-1 Composer, a unified framework designed to efficiently model marginal, conditional, and joint distributions over multi-track music using a single model. Building upon an audio latent diffusion model, JEN-1 Composer extends the versatility of multi-track music generation. We introduce a progressive curriculum training strategy, which gradually escalates the difficulty of training tasks while ensuring the model's generalization ability and facilitating smooth transitions between different scenarios. During inference, users can iteratively generate and select music tracks, thus incrementally composing entire musical pieces in accordance with the Human-AI co-composition workflow. Our approach demonstrates state-of-the-art performance in controllable and high-fidelity multi-track music synthesis, marking a significant advancement in interactive AI-assisted music creation. Our demo pages are available at www.jenmusic.ai/research.

Figures (3)

• Figure 1: The Human-AI co-composition workflow of JEN-1 Composer. JEN-1 Composer generates multiple music tracks based on user-provided text prompts (specifying genres, eras, rhythms, etc.) and optional audio feedback, where users can select, edit, or upload tracks. The human feedback guides the generation of target tracks, ensuring temporal alignment and musical coherence. The iterative process of human feedback and AI generation continues until a harmonious and cohesive musical piece is achieved.
• Figure 2: The U-Net architecture used in Jen-1.
• Figure 3: Illustration of three generation modes using independent timesteps as indicators. In Marginal Generation, non-target track latents are fixed as Gaussian noise (timestep $T$) to minimize their impact on the target track's latent. Conditional Generation designates a timestep of $0$ for a conditional track, guiding the target track's generation. Joint Generation synchronizes multiple target tracks by sharing the same timestep $t$, allowing for coordinated denoising from $T$ to $0$.