Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Differentiable Modal Synthesis for Physical Modeling of Planar String Sound and Motion Simulation

Jin Woo Lee, Jaehyun Park, Min Jun Choi, Kyogu Lee

TL;DR

A novel model for simulating the spatio-temporal motion of nonlinear strings, integrating modal synthesis and spectral modeling within a neural network framework is introduced, achieving superior accuracy in string motion simulation compared to existing baseline architectures.

Abstract

While significant advancements have been made in music generation and differentiable sound synthesis within machine learning and computer audition, the simulation of instrument vibration guided by physical laws has been underexplored. To address this gap, we introduce a novel model for simulating the spatio-temporal motion of nonlinear strings, integrating modal synthesis and spectral modeling within a neural network framework. Our model leverages physical properties and fundamental frequencies as inputs, outputting string states across time and space that solve the partial differential equation characterizing the nonlinear string. Empirical evaluations demonstrate that the proposed architecture achieves superior accuracy in string motion simulation compared to existing baseline architectures. The code and demo are available online.

Differentiable Modal Synthesis for Physical Modeling of Planar String Sound and Motion Simulation

TL;DR

A novel model for simulating the spatio-temporal motion of nonlinear strings, integrating modal synthesis and spectral modeling within a neural network framework is introduced, achieving superior accuracy in string motion simulation compared to existing baseline architectures.

Abstract

While significant advancements have been made in music generation and differentiable sound synthesis within machine learning and computer audition, the simulation of instrument vibration guided by physical laws has been underexplored. To address this gap, we introduce a novel model for simulating the spatio-temporal motion of nonlinear strings, integrating modal synthesis and spectral modeling within a neural network framework. Our model leverages physical properties and fundamental frequencies as inputs, outputting string states across time and space that solve the partial differential equation characterizing the nonlinear string. Empirical evaluations demonstrate that the proposed architecture achieves superior accuracy in string motion simulation compared to existing baseline architectures. The code and demo are available online.
Paper Structure (17 sections, 1 theorem, 16 equations, 9 figures, 5 tables)

This paper contains 17 sections, 1 theorem, 16 equations, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Physical Modeling of Musical String Instrument
  4. Differentiable Digital Signal Processing
  5. Differentiable Modal Synthesis for Physical Modeling (DMSP)
  6. Problem Statement
  7. Network Architecture
  8. Loss Function
  9. Experiments
  10. Experimental Setup
  11. Results
  12. Conclusion
  13. Proof of the Linear Damped Stiff String Solution
  14. Nonlinear Damped Stiff String Vibration
  15. Damping coefficient
  16. ...and 2 more sections

Key Result

Proposition 1

A solution to the damped linear stiff string model eqn:linear-wave with a clamped boundary condition $u(\pm L/2,t)=u_{x}(\pm L/2,t)=0$ and the initial condition given as $u(x,0)=u_0(x)$ and $\partial_t u(x,0)=0$ can be expressed as

Figures (9)

  • Figure 1: System overview. The DMSP model encodes the physical properties of a string (e.g., tension, stiffness, damping, and initial conditions) to estimate the displacement of the string plucked at pitch $f_0$ at a given time $t \in [0, \infty)$ and position $x \in \Omega$. By concatenating the DMSP outputs over the domain $(x, t) \in \Omega \times [0, \infty)$, the simulated motion of the string can be visualized. Reading the outputs at a particular position $x$ allows hearing the synthesized string sound, akin to listening with a stethoscope at the pickup position.
  • Figure 2: The planar string system.
  • Figure 3: Network architecture. DMSP synthesizes a pitch skeleton with an inharmonic structure, drawing upon overtones derived from the modes of the string. The modes can either be derived directly using the modal decomposition (DMSP-Hybrid, the hybrid of DMSP and Modal), or using the neural network trained to estimate the modes (DMSP, the fully-neural-network method). Yet, relying solely on modal frequencies and corresponding shape functions delineates a linear solution, which falls short of capturing the nuances of nonlinear string motion. To address this, DMSP introduces FM and AM blocks to modulate the modes of the linear solution. This modulation process enables DMSP to estimate the pitch skeleton of the nonlinear solution. Consequently, the output waveform is synthesized through the spectral modeling pipeline, incorporating both (in)harmonic components and the filtered noise.
  • Figure 4: Visualization of the string displacement over time (horizontal) and space (vertical). For different initial conditions, the results synthesized by DMSP are shown as solid black lines and those simulated by FDTD as dashed gray lines.
  • Figure 5: Objective scores over the change of physical parameters.
  • ...and 4 more figures

Theorems & Definitions (2)

  • Proposition 1
  • proof