Generation of Musical Timbres using a Text-Guided Diffusion Model

TL;DR

This work introduces a text-guided, diffusion-based framework for generating and manipulating musical timbres at the single-note level. By operating in a latent spectral space learned via VQ-GAN and aligning text descriptions with timbre representations through contrastive learning, the method jointly models magnitude and phase to produce realistic timbres conditioned on natural language prompts. It enables both note-level timbre synthesis and targeted timbre editing through RePaint-inspired spectral inpainting and global transformations, offering a musician-centric workflow that preserves creative input while expanding timbre fertile ground. Quantitative and qualitative evaluations on NSynth demonstrate improvements in realism, diversity, and text-timbre alignment, with practical implications for composers and performers seeking flexible, non-existent timbres for electronic instruments and DAWs.

Abstract

In recent years, text-to-audio systems have achieved remarkable success, enabling the generation of complete audio segments directly from text descriptions. While these systems also facilitate music creation, the element of human creativity and deliberate expression is often limited. In contrast, the present work allows composers, arrangers, and performers to create the basic building blocks for music creation: audio of individual musical notes for use in electronic instruments and DAWs. Through text prompts, the user can specify the timbre characteristics of the audio. We introduce a system that combines a latent diffusion model and multi-modal contrastive learning to generate musical timbres conditioned on text descriptions. By jointly generating the magnitude and phase of the spectrogram, our method eliminates the need for subsequently running a phase retrieval algorithm, as related methods do. Audio examples, source code, and a web app are available at https://wxuanyuan.github.io/Musical-Note-Generation/

Figures (6)

• Figure 1: Difference between text-to-sound systems (Top) and the proposed method (Bottom). Unlike end-to-end text-to-sound systems, human creativity in musical arrangement is preserved. Our method first generates fixed-length musical notes with the desired timbre based on text descriptions (\ref{['sec:note_generation']}). This generated note can optionally be modified by the user for more fine-grained control of desired output. After that, the fixed-length notes are adjusted to varying lengths using the diffusion-based inpainting method RePaint. All three stages use the same models, trained solely on fixed-length samples. Finally, these different notes are arranged by human musicians to create music.
• Figure 2: Architecture overview of our framework for generating timbre. It combines multi-modal contrastive learning and latent diffusion models. STFT+ and ISTFT+ represent the non-trainable time-frequency domain transformations of audio signals $S$. A pretrained LLM is used to augment labels such as "bright, guitar" from the NSynth dataset to diverse text descriptions. The training is divided into three phases: (1) A VQ-GAN (in yellow) is trained as an autoencoder for the spectral representation of real samples. Its discriminator $D$ is trained to distinguish spectral representations of real samples (i.e. $x$ for all training samples) from those of generated samples (i.e. $\hat{x}$ for all training samples). The encoder, decoder, and quantizer are trained to fool the discriminator, i.e. to produce realistic $\hat{x}$. (2) A text encoder (pretrained using CLAP CLAP) and a timbre encoder (both shown in green) are trained to map text descriptions and the timbre representation $\hat{z}$ into a unified embedding space via contrastive learning. (3) A diffusion model (in blue) is trained to produce latent representations conditioned by the text embeddings. During the inference stage, the output of the diffusion model is passed to the VQ-GAN decoder. Details of the individual components are provided in \ref{['sec:note_generation']}. For further details on the model components, including hyperparameters and training settings, please refer to the https://wxuanyuan.github.io/Musical-Note-Generation/.
• Figure 3: Results of conditioned sampling with varying guidance scales $w$. As $w$ increases, more high-frequency components are introduced into the spectrogram in line with the text description.
• Figure 4: Variations of the mean amplitude distribution along frequency (left) and time (right) dimensions at different guidance scales $w$. With the increase of guidance scale $w$, the amplitude distribution varies as required by the text description; specifically, "dark" for the lower frequency range, and "long release" for the release stage of the sound.
• Figure 5: Timbre inpainting examples. The modified regions are highlighted with masks bordered in light blue. The text description is an empty string.