Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Steering Autoregressive Music Generation with Recursive Feature Machines

Daniel Zhao, Daniel Beaglehole, Taylor Berg-Kirkpatrick, Julian McAuley, Zachary Novack

TL;DR

MusicRFM introduces a principled activation-space steering framework for autoregressive music generation by adapting Recursive Feature Machines (RFMs) to MusicGen. Lightweight RFM probes produce orthogonal, interpretable directions along which hidden activations can be steered in real time, without fine-tuning the base model. The method employs layer pruning, time-based schedules, and multi-direction control to balance musical controllability with audio fidelity, achieving substantial improvements in target-note generation while preserving text prompt adherence. Evaluations on synthetic (SynTheory) and real (MusicBench) data show strong controllability with moderate distributional drift, and the approach is accompanied by release-ready code for reproducibility and further exploration.

Abstract

Controllable music generation remains a significant challenge, with existing methods often requiring model retraining or introducing audible artifacts. We introduce MusicRFM, a framework that adapts Recursive Feature Machines (RFMs) to enable fine-grained, interpretable control over frozen, pre-trained music models by directly steering their internal activations. RFMs analyze a model's internal gradients to produce interpretable "concept directions", or specific axes in the activation space that correspond to musical attributes like notes or chords. We first train lightweight RFM probes to discover these directions within MusicGen's hidden states; then, during inference, we inject them back into the model to guide the generation process in real-time without per-step optimization. We present advanced mechanisms for this control, including dynamic, time-varying schedules and methods for the simultaneous enforcement of multiple musical properties. Our method successfully navigates the trade-off between control and generation quality: we can increase the accuracy of generating a target musical note from 0.23 to 0.82, while text prompt adherence remains within approximately 0.02 of the unsteered baseline, demonstrating effective control with minimal impact on prompt fidelity. We release code to encourage further exploration on RFMs in the music domain.

Steering Autoregressive Music Generation with Recursive Feature Machines

TL;DR

MusicRFM introduces a principled activation-space steering framework for autoregressive music generation by adapting Recursive Feature Machines (RFMs) to MusicGen. Lightweight RFM probes produce orthogonal, interpretable directions along which hidden activations can be steered in real time, without fine-tuning the base model. The method employs layer pruning, time-based schedules, and multi-direction control to balance musical controllability with audio fidelity, achieving substantial improvements in target-note generation while preserving text prompt adherence. Evaluations on synthetic (SynTheory) and real (MusicBench) data show strong controllability with moderate distributional drift, and the approach is accompanied by release-ready code for reproducibility and further exploration.

Abstract

Controllable music generation remains a significant challenge, with existing methods often requiring model retraining or introducing audible artifacts. We introduce MusicRFM, a framework that adapts Recursive Feature Machines (RFMs) to enable fine-grained, interpretable control over frozen, pre-trained music models by directly steering their internal activations. RFMs analyze a model's internal gradients to produce interpretable "concept directions", or specific axes in the activation space that correspond to musical attributes like notes or chords. We first train lightweight RFM probes to discover these directions within MusicGen's hidden states; then, during inference, we inject them back into the model to guide the generation process in real-time without per-step optimization. We present advanced mechanisms for this control, including dynamic, time-varying schedules and methods for the simultaneous enforcement of multiple musical properties. Our method successfully navigates the trade-off between control and generation quality: we can increase the accuracy of generating a target musical note from 0.23 to 0.82, while text prompt adherence remains within approximately 0.02 of the unsteered baseline, demonstrating effective control with minimal impact on prompt fidelity. We release code to encourage further exploration on RFMs in the music domain.
Paper Structure (43 sections, 13 equations, 2 figures, 10 tables)

This paper contains 43 sections, 13 equations, 2 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Controllable Music and Audio Generation
  4. Activation-Level Steering in Generative Models
  5. Recursive Feature Machines (RFMs)
  6. Methods
  7. Background on Recursive Feature Machines
  8. MusicRFM: RFM Steering for Music Generation
  9. Synthetic Dataset for Probe Training.
  10. Feature Extraction.
  11. Probe training and steering.
  12. Improving Robustness in Audio-Domain Steering
  13. Layer Pruning
  14. Time-Control Schedules
  15. Multi-Direction and Staggered Control
  16. ...and 28 more sections

Figures (2)

  • Figure 1: Time-based steering analyses. (a) Probe softmax follows prescribed schedules faithfully. (b) Crossfade experiments show expected decay–rise patterns between two target notes.
  • Figure 2: Single-direction steering metrics as a function of control coefficient $\eta_0$. Top Left: Fréchet Distance (FD; $\downarrow$) increases with stronger control. Top Right: Maximum Mean Discrepancy (MMD; $\downarrow$) shows a similar trend. Bottom Left: CLAP alignment ($\uparrow$) to the text prompt remains relatively stable for most categories. Bottom Right: Probe accuracy shows that, despite poor performance on generated data, there is an upwards trend in accuracy as we increase the control coef $\eta_0$.