FlowSynth: Instrument Generation Through Distributional Flow Matching and Test-Time Search

TL;DR

FlowSynth addresses timbre drift in virtual instrument generation by introducing Distributional Flow Matching (DFM), which models the velocity field as a Gaussian distribution and trains via negative log-likelihood to capture predictive uncertainty. At test time, uncertainty-guided sampling and a principled search over multiple trajectories enable timbre-consistent outputs across pitches, with a temperature schedule that scales with compute budget. Built on a Diffusion Transformer backbone with CLAP and a DiffRhythm VAE, FlowSynth achieves state-of-the-art results over TokenSynth, and its performance scales effectively with increased search budgets. The approach offers a practical path to professional-quality AI-generated instruments suitable for real-time performance, especially as training data and compute expand.

Abstract

Virtual instrument generation requires maintaining consistent timbre across different pitches and velocities, a challenge that existing note-level models struggle to address. We present FlowSynth, which combines distributional flow matching (DFM) with test-time optimization for high-quality instrument synthesis. Unlike standard flow matching that learns deterministic mappings, DFM parameterizes the velocity field as a Gaussian distribution and optimizes via negative log-likelihood, enabling the model to express uncertainty in its predictions. This probabilistic formulation allows principled test-time search: we sample multiple trajectories weighted by model confidence and select outputs that maximize timbre consistency. FlowSynth outperforms the current state-of-the-art TokenSynth baseline in both single-note quality and cross-note consistency. Our approach demonstrates that modeling predictive uncertainty in flow matching, combined with music-specific consistency objectives, provides an effective path to professional-quality virtual instruments suitable for real-time performance.

Figures (2)

• Figure 1: Proposed distributional flow matching (DFM) architecture. (Left) Training phase: The diffusion transformer (DiT) learns to predict the instantaneous velocity field by projecting latent representations into mean ($\mu$) and variance ($\sigma$) parameters of a Gaussian distribution. Negative log-likelihood (NLL) loss is minimized between the ground-truth latent at current timestep and this learned distribution. (Right) Inference phase: Multiple velocity field samples are drawn from the learned Gaussian distribution at each generation timestep $t$. These samples undergo iterative one-step generation until the final timestep $T$, where a search metric evaluates all candidate latents to select the optimal output for the current timestep.
• Figure 2: Sample CLAP score v.s. guided sampling steps