Pay (Cross) Attention to the Melody: Curriculum Masking for Single-Encoder Melodic Harmonization

TL;DR

This work tackles melodic harmonization with a single‑encoder transformer and identifies weak melody–harmony cross‑attention in prior training strategies. It introduces a full‑to‑full (FF) curriculum that starts with all harmony tokens masked and gradually unmasks them to force early melody conditioning, improving cross‑modal integration. Across HookTheory and jazz standards, FF yields consistent gains in chord progression quality, melody–harmony alignment, and rhythmic coherence, especially in out‑of‑domain settings, with quarter‑note quantization and bar‑token integration further boosting performance. The results highlight the importance of curriculum design for robust melodic conditioning and point to practical benefits for controllable, interactive melodic harmonization systems.

Abstract

Melodic harmonization, the task of generating harmonic accompaniments for a given melody, remains a central challenge in computational music generation. Recent single encoder transformer approaches have framed harmonization as a masked sequence modeling problem, but existing training curricula inspired by discrete diffusion often result in weak (cross) attention between melody and harmony. This leads to limited exploitation of melodic cues, particularly in out-of-domain contexts. In this work, we introduce a training curriculum, FF (full-to-full), which keeps all harmony tokens masked for several training steps before progressively unmasking entire sequences during training to strengthen melody-harmony interactions. We systematically evaluate this approach against prior curricula across multiple experimental axes, including temporal quantization (quarter vs. sixteenth note), bar-level vs. time-signature conditioning, melody representation (full range vs. pitch class), and inference-time unmasking strategies. Models are trained on the HookTheory dataset and evaluated both in-domain and on a curated collection of jazz standards, using a comprehensive set of metrics that assess chord progression structure, harmony-melody alignment, and rhythmic coherence. Results demonstrate that the proposed FF curriculum consistently outperforms baselines in nearly all metrics, with particularly strong gains in out-of-domain evaluations where harmonic adaptability to novel melodic queues is crucial. We further find that quarter-note quantization, intertwining of bar tokens, and pitch-class melody representations are advantageous in the FF setting. Our findings highlight the importance of training curricula in enabling effective melody conditioning and suggest that full-to-full unmasking offers a robust strategy for single encoder harmonization.

Figures (3)

• Figure 1: Overview of the single-encoder architecture for melodic harmonization. Left: abstract representation of the transformer encoder input (melody in the first half, harmony in the second) and its output. Right: schematic of the encoder’s attention map. When predicting a harmony token at time step $t$ ($h_t$), only the lower half of the map is functionally relevant: the lower-left quadrant captures harmony–given–melody interactions ("cross" attention), while the lower-right quadrant captures harmony–given–harmony interactions ("self" attention).
• Figure 2: Average attention maps across all layers and heads for the training methods from prior work (a, b) and for the proposed method (c). In (a) and (b), cross-attention from harmony to melody is largely absent, as indicated by the lack of diagonal patterns in the lower-left quadrant. In contrast, (c) exhibits a distinct diagonal structure, highlighting effective melody-to-harmony cross-attention.
• Figure 3: Quarter-note resolution of pitch-class pianoroll with integrated bar information. Melody is represented as a $13 \times T$ matrix, with the extra row marking barline positions. Harmony is shown as a parallel sequence of chord tokens with inserted <bar> tokens. A random segment of the artificial dataset described in Section \ref{['subsec:problem']} is depicted.