Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Music Proofreading with RefinPaint: Where and How to Modify Compositions given Context

Pedro Ramoneda, Martin Rocamora, Taketo Akama

TL;DR

RefinPaint is an iterative technique that improves the sampling process by identifying the weaker music elements using a feedback model, which then informs the choices for resampling by an inpainting model.

Abstract

Autoregressive generative transformers are key in music generation, producing coherent compositions but facing challenges in human-machine collaboration. We propose RefinPaint, an iterative technique that improves the sampling process. It does this by identifying the weaker music elements using a feedback model, which then informs the choices for resampling by an inpainting model. This dual-focus methodology not only facilitates the machine's ability to improve its automatic inpainting generation through repeated cycles but also offers a valuable tool for humans seeking to refine their compositions with automatic proofreading. Experimental results suggest RefinPaint's effectiveness in inpainting and proofreading tasks, demonstrating its value for refining music created by both machines and humans. This approach not only facilitates creativity but also aids amateur composers in improving their work.

Music Proofreading with RefinPaint: Where and How to Modify Compositions given Context

TL;DR

RefinPaint is an iterative technique that improves the sampling process by identifying the weaker music elements using a feedback model, which then informs the choices for resampling by an inpainting model.

Abstract

Autoregressive generative transformers are key in music generation, producing coherent compositions but facing challenges in human-machine collaboration. We propose RefinPaint, an iterative technique that improves the sampling process. It does this by identifying the weaker music elements using a feedback model, which then informs the choices for resampling by an inpainting model. This dual-focus methodology not only facilitates the machine's ability to improve its automatic inpainting generation through repeated cycles but also offers a valuable tool for humans seeking to refine their compositions with automatic proofreading. Experimental results suggest RefinPaint's effectiveness in inpainting and proofreading tasks, demonstrating its value for refining music created by both machines and humans. This approach not only facilitates creativity but also aids amateur composers in improving their work.
Paper Structure (18 sections, 1 equation, 6 figures, 3 tables, 3 algorithms)

This paper contains 18 sections, 1 equation, 6 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Methodology
  3. Inpainting model ($\mathcal{I}$)
  4. Training the Inpainting Model ($\mathcal{I}$)
  5. Feedback model ($\mathcal{F}$)
  6. Training the Feedback model ($\mathcal{F}$)
  7. Generation of MIDI sequences (RefinPaint)
  8. Related work
  9. Experimental Setup
  10. Data preparation
  11. Model development
  12. Inpainting results
  13. Divide and conquer with the inpainting model
  14. Objective evaluation of proofreading inpainting
  15. Listening test of proofreading inpainting
  16. ...and 3 more sections

Figures (6)

  • Figure 1: A user selects a MIDI section for enhancement (gray rectangle). Our methodology uses token-level feedback (blue) to highlight critical notes or sequences (red) for regeneration (green). This cycle repeats iteratively.
  • Figure 2: Encoder-decoder architecture for inpainting, given a user-provided mask $M_u$ with a subset mask $M_s$.
  • Figure 3: The Feedback algorithm identifies the most realistic tokens by training it to discern between real and synthetic music tokens.
  • Figure 4: RefinPaint uses inpainting and feedback models to iteratively suggest changes, based on specific note feedback. It reduces the selected tokens in each iteration.
  • Figure 5: Results of the participants' votes for the listening test comparing PIA and RefinPaint (Ours) along different fragment sizes (50%, 30%, and 10%).
  • ...and 1 more figures