Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Text2midi-InferAlign: Improving Symbolic Music Generation with Inference-Time Alignment

Abhinaba Roy, Geeta Puri, Dorien Herremans

TL;DR

Text2midi-InferAlign addresses the challenge of aligning symbolic music generation with textual captions without retraining existing models. It introduces a reward-guided tree search at inference time, guided by two objectives: text-audio consistency and harmonic consistency, with caption mutations to explore a broader caption space and reward-guided beam replacements to exploit high-reward states. The composite reward R(s,x) = α Ra + β Rh combines CLAP-based caption-audio similarity and key-compatibility measures to improve musicality and caption adherence. Evaluations on MidiCaps show improvements across objective metrics (e.g., CLAP, tempo-related measures, and key accuracy) and subjective listening tests, highlighting the method’s potential for producing more coherent, caption-consistent MIDI outputs without additional training.

Abstract

We present Text2midi-InferAlign, a novel technique for improving symbolic music generation at inference time. Our method leverages text-to-audio alignment and music structural alignment rewards during inference to encourage the generated music to be consistent with the input caption. Specifically, we introduce two objectives scores: a text-audio consistency score that measures rhythmic alignment between the generated music and the original text caption, and a harmonic consistency score that penalizes generated music containing notes inconsistent with the key. By optimizing these alignment-based objectives during the generation process, our model produces symbolic music that is more closely tied to the input captions, thereby improving the overall quality and coherence of the generated compositions. Our approach can extend any existing autoregressive model without requiring further training or fine-tuning. We evaluate our work on top of Text2midi - an existing text-to-midi generation model, demonstrating significant improvements in both objective and subjective evaluation metrics.

Text2midi-InferAlign: Improving Symbolic Music Generation with Inference-Time Alignment

TL;DR

Text2midi-InferAlign addresses the challenge of aligning symbolic music generation with textual captions without retraining existing models. It introduces a reward-guided tree search at inference time, guided by two objectives: text-audio consistency and harmonic consistency, with caption mutations to explore a broader caption space and reward-guided beam replacements to exploit high-reward states. The composite reward R(s,x) = α Ra + β Rh combines CLAP-based caption-audio similarity and key-compatibility measures to improve musicality and caption adherence. Evaluations on MidiCaps show improvements across objective metrics (e.g., CLAP, tempo-related measures, and key accuracy) and subjective listening tests, highlighting the method’s potential for producing more coherent, caption-consistent MIDI outputs without additional training.

Abstract

We present Text2midi-InferAlign, a novel technique for improving symbolic music generation at inference time. Our method leverages text-to-audio alignment and music structural alignment rewards during inference to encourage the generated music to be consistent with the input caption. Specifically, we introduce two objectives scores: a text-audio consistency score that measures rhythmic alignment between the generated music and the original text caption, and a harmonic consistency score that penalizes generated music containing notes inconsistent with the key. By optimizing these alignment-based objectives during the generation process, our model produces symbolic music that is more closely tied to the input captions, thereby improving the overall quality and coherence of the generated compositions. Our approach can extend any existing autoregressive model without requiring further training or fine-tuning. We evaluate our work on top of Text2midi - an existing text-to-midi generation model, demonstrating significant improvements in both objective and subjective evaluation metrics.
Paper Structure (19 sections, 6 equations, 1 figure, 5 tables, 1 algorithm)

This paper contains 19 sections, 6 equations, 1 figure, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Objectives and Reward Functions
  5. Exploration
  6. Exploitation
  7. Experimental Setup
  8. Baseline & Test set
  9. Objective Evaluation Metrics
  10. Compression ratio
  11. CLAP
  12. Feature wise comparison
  13. Listening Test
  14. Implementatinal Details
  15. Results
  16. ...and 4 more sections

Figures (1)

  • Figure 1: Overview of Text2midi-InferAlign, using Text2midi for inference and reward score(s) for alignment.