Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Depth-Structured Music Recurrence: Budgeted Recurrent Attention for Full-Piece Symbolic Music Modeling

Yungang Yi

TL;DR

Depth-Structured Music Recurrence (DSMR) is introduced, a recurrent long-context Transformer for full-piece symbolic music modeling that extends context beyond fixed-length excerpts via segment-level recurrence with detached cross-segment states, featuring a layer-wise memory-horizon schedule that budgets recurrent KV states across depth.

Abstract

Long-context modeling is essential for symbolic music generation, since motif repetition and developmental variation can span thousands of musical events. However, practical composition and performance workflows frequently rely on resource-limited devices (e.g., electronic instruments and portable computers), making heavy memory and attention computation difficult to deploy. We introduce Depth-Structured Music Recurrence (DSMR), a recurrent long-context Transformer for full-piece symbolic music modeling that extends context beyond fixed-length excerpts via segment-level recurrence with detached cross-segment states, featuring a layer-wise memory-horizon schedule that budgets recurrent KV states across depth. DSMR is trained in a single left-to-right pass over each complete composition, akin to how a musician experiences it from beginning to end, while carrying recurrent cross-segment states forward. Within this recurrent framework, we systematically study how depth-wise horizon allocations affect optimization, best-checkpoint perplexity, and efficiency. By allocating different history-window lengths across layers while keeping the total recurrent-state budget fixed, DSMR creates depth-dependent temporal receptive fields within a recurrent attention stack without reducing compute depth. Our main instantiation is a two-scale DSMR schedule that allocates long history windows to lower layers and a uniform short window to the remaining layers. Experiments on the piano performance dataset MAESTRO demonstrate that two-scale DSMR provides a practical quality--efficiency recipe for full-length long-context symbolic music modeling with recurrent attention under limited computational resources.

Depth-Structured Music Recurrence: Budgeted Recurrent Attention for Full-Piece Symbolic Music Modeling

TL;DR

Depth-Structured Music Recurrence (DSMR) is introduced, a recurrent long-context Transformer for full-piece symbolic music modeling that extends context beyond fixed-length excerpts via segment-level recurrence with detached cross-segment states, featuring a layer-wise memory-horizon schedule that budgets recurrent KV states across depth.

Abstract

Long-context modeling is essential for symbolic music generation, since motif repetition and developmental variation can span thousands of musical events. However, practical composition and performance workflows frequently rely on resource-limited devices (e.g., electronic instruments and portable computers), making heavy memory and attention computation difficult to deploy. We introduce Depth-Structured Music Recurrence (DSMR), a recurrent long-context Transformer for full-piece symbolic music modeling that extends context beyond fixed-length excerpts via segment-level recurrence with detached cross-segment states, featuring a layer-wise memory-horizon schedule that budgets recurrent KV states across depth. DSMR is trained in a single left-to-right pass over each complete composition, akin to how a musician experiences it from beginning to end, while carrying recurrent cross-segment states forward. Within this recurrent framework, we systematically study how depth-wise horizon allocations affect optimization, best-checkpoint perplexity, and efficiency. By allocating different history-window lengths across layers while keeping the total recurrent-state budget fixed, DSMR creates depth-dependent temporal receptive fields within a recurrent attention stack without reducing compute depth. Our main instantiation is a two-scale DSMR schedule that allocates long history windows to lower layers and a uniform short window to the remaining layers. Experiments on the piano performance dataset MAESTRO demonstrate that two-scale DSMR provides a practical quality--efficiency recipe for full-length long-context symbolic music modeling with recurrent attention under limited computational resources.
Paper Structure (40 sections, 16 equations, 11 figures, 2 tables)

This paper contains 40 sections, 16 equations, 11 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. From Quadratic Attention to Subquadratic Long-Context Mechanisms
  4. Recurrent Transformers and Budgeted Stateful Memory
  5. Layer-Wise Memory Optimization
  6. Long-Context Symbolic Music Modeling under Resource Constraints
  7. Method
  8. DSMR's recurrent backbone
  9. Training-time sampling and segmentation
  10. Full-attention approximation of a vanilla Transformer as a reference model
  11. Depth-Structured Music Recurrence (DSMR) with layer-wise horizon
  12. Two-scale DSMR: long horizons in lower layers + a uniform short horizon elsewhere
  13. Gate-guided layer selection as a comparison variant
  14. Experimental Setup
  15. Datasets and tokenization
  16. ...and 25 more sections

Figures (11)

  • Figure 1: Full-piece segment-level recurrence. Attention for the current segment (green) is computed using its own tokens together with cached key/value states from all previous history (red). The history horizon is managed on a layer-wise basis; in our two-scale DSMR schedule, some memory is discarded (yellow).
  • Figure 2: Depth-Structured Music Recurrence (DSMR) in a recurrent layer. Key/value states from previous segments flow forward as a history buffer and are truncated by a layer-specific horizon to satisfy a fixed memory budget. The retained history is concatenated with the current segment’s key/value context to form the attention source for that layer.
  • Figure 3: Layer-wise memory-horizon schedules (allocations) for DSMR variants.
  • Figure 4: Best validation perplexity (PPL) versus peak GPU memory across DSMR variants, with full attention as a reference.
  • Figure 5: Gate statistics evolution (without STE), showing overall shrinkage and increasing differentiation of gates across depth.
  • ...and 6 more figures