SMITIN: Self-Monitored Inference-Time INtervention for Generative Music Transformers

TL;DR

SMITIN introduces a self-monitored inference-time intervention framework that steers a pre-trained music transformer by training per-head classifier probes to detect target musical traits and applying head-specific interventions. A self-monitoring loop dynamically modulates intervention strength to balance trait incorporation with musical coherence, with soft-weighting and automated head-selection to avoid manual tuning. Evaluations on audio continuation and text-to-music tasks show improved control over instrument addition while preserving audio quality and distributional realism, supported by objective metrics and subjective listening tests. Ablations demonstrate robustness to probe data size and direction choice, and visualization analyses reveal meaningful head-level representations underpinning controllability, offering practical knobs for musicians to guide generation without retraining. Overall, SMITIN provides fine-grained, real-time control of large generative music models, enabling targeted musical traits to be added or removed with minimal loss of realism.

Abstract

We introduce Self-Monitored Inference-Time INtervention (SMITIN), an approach for controlling an autoregressive generative music transformer using classifier probes. These simple logistic regression probes are trained on the output of each attention head in the transformer using a small dataset of audio examples both exhibiting and missing a specific musical trait (e.g., the presence/absence of drums, or real/synthetic music). We then steer the attention heads in the probe direction, ensuring the generative model output captures the desired musical trait. Additionally, we monitor the probe output to avoid adding an excessive amount of intervention into the autoregressive generation, which could lead to temporally incoherent music. We validate our results objectively and subjectively for both audio continuation and text-to-music applications, demonstrating the ability to add controls to large generative models for which retraining or even fine-tuning is impractical for most musicians. Audio samples of the proposed intervention approach are available on our demo page http://tinyurl.com/smitin .

Figures (6)

• Figure 1: Overall pipeline of SMITIN for inference-time intervention on a pre-trained music generative transformer. The process attempts to enforce specific musical factors (e.g., presence of a particular instrument) during the generation process. SMITIN utilizes a self-monitoring technique to dynamically adjust the intervention strength at each generation step, enabling precise control over the inclusion of the target characteristic while preserving the musical integrity of the output.
• Figure 2: Overview of probing MusicGen. Each audio sample in a labeled dataset is converted to EnCodec tokens and input into MusicGen to predict the next token. The activations for the last time step (orange dots) for each attention head in each layer (blue dots) are used to train a logistic regression classifier (probe).
• Figure 3: Instrument recognition performance of individual attention head probes from the MusicGen$_\text{large}$ model activations, sorted by accuracy, with all colorbars normalized to the same range. The values in brackets indicate the highest accuracy of the probe classifier for each respective instrument task, followed by the threshold value $\tau$, which is defined in Section \ref{['subsec:self_monitoring']}.
• Figure 4: Inferred prediction of the top-K probes' monitored decision along the time axis. The yellow line, green line, and the shaded blue region denote the median, mean, and standard deviation of inferred outputs by the probes, respectively. The red dashed line indicates threshold value $\tau$ of the current monitoring probes. (Left) Monitored result on a real-world music sample. The high prediction (close to 1.0) until 3.5 seconds reflects the actual presence of drums, which aligns with the audio sample where drums are present only up to that point. (Right) The four sub-plots display the results of audio continuation on the same input music with varying ITI frequencies ($s=[1, 5, 10, 20]$). These illustrate that more frequent intervention leads to a swifter convergence towards the target musical factor, at the expense of losing musical consistency with the input music.
• Figure 5: Temporal dynamics of maintaining "realism" in audio continuation of real music sample. The graph tracks the probability of generated music being classified as $\langle$real$\rangle$ over time, where all configurations of SMITIN demonstrate an enhanced capacity to preserve realistic music qualities over time.