BandCondiNet: Parallel Transformers-based Conditional Popular Music Generation with Multi-View Features

TL;DR

BandCondiNet advances conditional multitrack popular music generation by introducing 3D multi-view features that encode track- and bar-level information, and by employing a parallel Transformer architecture with Structure Enhanced Attention (SEA) and Cross-Track Transformer (CTT) to strengthen musical structure and inter-track harmony. The model encodes per-track condition embeddings, uses per-track Bottom and Top Decoders augmented with SEA, and fuses cross-instrument bar-level tokens via CTT, enabling robust long-sequence generation. Objective results show BandCondiNet outperforms baselines on fidelity and speed, especially on longer sequences, while subjective evaluations confirm superior richness and overall quality, particularly for 64-bar pieces. Overall, the approach offers a scalable, high-fidelity framework for conditional multitrack music generation with strong structural coherence and cross-track harmony, with future work aimed at flexible instrumentation and user-friendly input interfaces.

Abstract

Conditional music generation offers significant advantages in terms of user convenience and control, presenting great potential in AI-generated content research. However, building conditional generative systems for multitrack popular songs presents three primary challenges: insufficient fidelity of input conditions, poor structural modeling, and inadequate inter-track harmony learning in generative models. To address these issues, we propose BandCondiNet, a conditional model based on parallel Transformers, designed to process the multiple music sequences and generate high-quality multitrack samples. Specifically, we propose multi-view features across time and instruments as high-fidelity conditions. Moreover, we propose two specialized modules for BandCondiNet: Structure Enhanced Attention (SEA) to strengthen the musical structure, and Cross-Track Transformer (CTT) to enhance inter-track harmony. We conducted both objective and subjective evaluations on two popular music datasets with different sequence lengths. Objective results on the shorter dataset show that BandCondiNet outperforms other conditional models in 9 out of 10 metrics related to fidelity and inference speed, with the exception of Chord Accuracy. On the longer dataset, BandCondiNet surpasses all conditional models across all 10 metrics. Subjective evaluations across four criteria reveal that BandCondiNet trained on the shorter dataset performs best in Richness and performs comparably to state-of-the-art models in the other three criteria, while significantly outperforming them across all criteria when trained on the longer dataset. To further expand the application scope of BandCondiNet, future work should focus on developing an advanced conditional model capable of adapting to more user-friendly input conditions and supporting flexible instrumentation.

Figures (6)

• Figure 1: The overall architecture of BandCondiNet. BandCondiNet is designed based on the canonical Sequence-to-Sequence framework and contains Feature Encoders, Bottom Decoders, Cross-Track Transformer (CTT), and Top Decoders. The Structure Enhanced Attention (SEA) module is utilized both in Bottom Decoders and the Top Decoders.
• Figure 2: A visual comparison of various conditions used in conditional music generation: on the left are global conditions, in the middle are the commonly adopted fine-grained conditions, and on the right are our proposed multi-view features.
• Figure 3: The expansion operation of the SE-SA module consists of three steps: 1) calculating a single score for each bar pairs, 2) tiling this score across all tokens within that bar, and 3) populating the expanded score matrix with these tiled values. $B$ denotes the bar numbers, and $T$ represents the length of token sequences.
• Figure 4: Grouping and Updating operations of Cross-Track Transformer. $B$ denotes the bar numbers, $I$ denotes track numbers, and $d$ represents the hidden size.
• Figure 5: Subjective evaluation for samples generated by BandCondiNet and the four benchmark models. The mark of NS between the two bars represents that there is no significant difference ($p\geq 0.05$ with the one-tailed $t$-test) between the two values.