Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HARP 2.0: Expanding Hosted, Asynchronous, Remote Processing for Deep Learning in the DAW

Christodoulos Benetatos, Frank Cwitkowitz, Nathan Pruyne, Hugo Flores Garcia, Patrick O'Reilly, Zhiyao Duan, Bryan Pardo

TL;DR

The paper addresses the barrier for DAW users to access cutting-edge deep learning audio models by providing hosted, asynchronous processing via Gradio endpoints inside a plugin. HARP 2.0 introduces MIDI support, time-stamped labeling visualization, and a streamlined pyharp API, plus interface and stability enhancements. Contributions include cross-platform support, a curl-based Gradio integration, and an overhaul of the interface, enabling audio, MIDI, and labeling workflows entirely within the DAW. The work aims to bridge model development and music creation by making DL models more accessible and integrable into standard DAW workflows, with open-source release.

Abstract

HARP 2.0 brings deep learning models to digital audio workstation (DAW) software through hosted, asynchronous, remote processing, allowing users to route audio from a plug-in interface through any compatible Gradio endpoint to perform arbitrary transformations. HARP renders endpoint-defined controls and processed audio in-plugin, meaning users can explore a variety of cutting-edge deep learning models without ever leaving the DAW. In the 2.0 release we introduce support for MIDI-based models and audio/MIDI labeling models, provide a streamlined pyharp Python API for model developers, and implement numerous interface and stability improvements. Through this work, we hope to bridge the gap between model developers and creatives, improving access to deep learning models by seamlessly integrating them into DAW workflows.

HARP 2.0: Expanding Hosted, Asynchronous, Remote Processing for Deep Learning in the DAW

TL;DR

The paper addresses the barrier for DAW users to access cutting-edge deep learning audio models by providing hosted, asynchronous processing via Gradio endpoints inside a plugin. HARP 2.0 introduces MIDI support, time-stamped labeling visualization, and a streamlined pyharp API, plus interface and stability enhancements. Contributions include cross-platform support, a curl-based Gradio integration, and an overhaul of the interface, enabling audio, MIDI, and labeling workflows entirely within the DAW. The work aims to bridge model development and music creation by making DL models more accessible and integrable into standard DAW workflows, with open-source release.

Abstract

HARP 2.0 brings deep learning models to digital audio workstation (DAW) software through hosted, asynchronous, remote processing, allowing users to route audio from a plug-in interface through any compatible Gradio endpoint to perform arbitrary transformations. HARP renders endpoint-defined controls and processed audio in-plugin, meaning users can explore a variety of cutting-edge deep learning models without ever leaving the DAW. In the 2.0 release we introduce support for MIDI-based models and audio/MIDI labeling models, provide a streamlined pyharp Python API for model developers, and implement numerous interface and stability improvements. Through this work, we hope to bridge the gap between model developers and creatives, improving access to deep learning models by seamlessly integrating them into DAW workflows.

Paper Structure

This paper contains 6 sections, 3 figures.

Table of Contents

  1. Introduction
  2. MIDI Support
  3. Labeling Support
  4. Interface
  5. Usability & Stability
  6. Conclusion

Figures (3)

  • Figure 1: The HARP 2.0 interface. Users can enter a path to a compatible Gradio endpoint or select from a menu of default endpoints (a); upon loading (b), the endpoint's interface (c) is displayed to the user, who can tweak controls before processing (d). Outputs are then rendered for playback and visualization.
  • Figure 2: (a) Piano roll for display and playback of MIDI files in HARP. (b) Audio-to-labels in HARP. Labels are displayed in multiple colors to denote audio similarity as detected by different embeddings. Details about the label appear in the bottom left 'info' box.
  • Figure :