Generalizing the SINDy approach with nested neural networks

TL;DR

Nested SINDy generalizes the SINDy framework by inserting polynomial and radial layers to capture compositions and products of basis functions, enabling symbolic regression for more complex dynamical systems. It introduces two architectures, PR and PRP, and demonstrates their effectiveness in function discovery and autonomous ODE discovery across multiple test cases, while also addressing sparsity via regularization and pruning. The approach achieves accurate symbolic approximations and sparse representations in simple trig-based targets and in nonlinear growth models (e.g., Gompertz), but it also reveals optimization challenges, necessitating careful hyperparameter initialization and training strategies. The findings suggest substantial potential for data-driven symbolic modeling, with future work focusing on robust optimization, automatic dictionary selection, and extensions to PDEs and spatially extended systems.

Abstract

Symbolic Regression (SR) is a widely studied field of research that aims to infer symbolic expressions from data. A popular approach for SR is the Sparse Identification of Nonlinear Dynamical Systems (SINDy) framework, which uses sparse regression to identify governing equations from data. This study introduces an enhanced method, Nested SINDy, that aims to increase the expressivity of the SINDy approach thanks to a nested structure. Indeed, traditional symbolic regression and system identification methods often fail with complex systems that cannot be easily described analytically. Nested SINDy builds on the SINDy framework by introducing additional layers before and after the core SINDy layer. This allows the method to identify symbolic representations for a wider range of systems, including those with compositions and products of functions. We demonstrate the ability of the Nested SINDy approach to accurately find symbolic expressions for simple systems, such as basic trigonometric functions, and sparse (false but accurate) analytical representations for more complex systems. Our results highlight Nested SINDy's potential as a tool for symbolic regression, surpassing the traditional SINDy approach in terms of expressivity. However, we also note the challenges in the optimization process for Nested SINDy and suggest future research directions, including the designing of a more robust methodology for the optimization process. This study proves that Nested SINDy can effectively discover symbolic representations of dynamical systems from data, offering new opportunities for understanding complex systems through data-driven methods.

Figures (11)

• Figure 1: Structure of the PR model for $d=2$ and 2 input variables.
• Figure 2: Structure of the PRP model for $k=2$, $d=2$, and 2 input dimensions.
• Figure 3: Case 1 from \ref{['sec:function_case_1']}: evolution of the learned function $x \mapsto \cos(x^2)$ over successive training epochs. Each subfigure represents the PR model's predictions (in blue) against the target function (in orange) at epochs 10, 70, 120, and 180, showcasing the model's progressive learning and convergence towards approximation of the target function. We observe that the first epochs learn on the interval $[0, 3]$, while subsequent epochs are able to generalize.
• Figure 4: Case 2 from \ref{['sec:function_case_2']}: comparison of the results from the PRP-Nested SINDy, which yielded the function $x \mapsto \mathop{\newline}\! \mathop{\mathrm{ {\operator@font sin}\newline}}\nolimits(x^2+1.57)$, with the target function $x \mapsto \cos(x^2)$, over the interval [0, 3].
• Figure 5: Case 3: comparison of the true and predicted models. A visual representation of the target function $(x,y) \mapsto 2 \mathop{\newline}\! \mathop{\mathrm{ {\operator@font sin}\newline}}\nolimits(x)\cos(y)$ and its learned approximation by the PRP model, given by \ref{['eq:prp_2D_result']}, over the specified domain.