Optimizing Hard Thresholding for Sparse Model Discovery

TL;DR

This work focuses on two approaches to the optimization of model selection algorithms, SINDy and an alternative using hard thresholding pursuit, and sees in both cases that annealing can improve model accuracy.

Abstract

Many model selection algorithms rely on sparse dictionary learning to provide interpretable and physics-based governing equations. The optimization algorithms typically use a hard thresholding process to enforce sparse activations in the model coefficients by removing library elements from consideration. By introducing an annealing scheme that reactivates a fraction of the removed terms with a cooling schedule, we are able to improve the performance of these sparse learning algorithms. We concentrate on two approaches to the optimization, SINDy, and an alternative using hard thresholding pursuit. We see in both cases that annealing can improve model accuracy. The effectiveness of annealing is demonstrated through comparisons on several nonlinear systems pulled from convective flows, excitable systems, and population dynamics. Finally we apply these algorithms to experimental data for projectile motion.

Figures (9)

• Figure 1: An example plot of the Van der Pol Oscillator with additive zero mean Gaussian noise with variance $0.01$ and a reconstructed solution using SHTreP-A.
• Figure 2: $\ell^1$ coefficient error on 300 different noise profiles for the Lorenz system with the width of the shaded portions being a standard deviation. The blue is SHTreP, the orange is SHTreP-A, the green is standard SINDy, and the purple region is SINDy-Anne. For SINDy, we assume a thresholding parameter for each of the equations: $\lambda_x = 0.4$, $\lambda_y = 0.6$, and $\lambda_z = 0.2$ and for the SHTreP algorithms, we choose a sparsity $s = 15$.
• Figure 3: $\ell^1$ error on 300 different noise profiles for the FitzHugh--Nagumo system. The colors are explained in Figure \ref{['fig: lorenz_noise_var']}. This uses $\lambda = 0.025$ for both $\dot{x}$ and $\dot{y}$ and $s = 6$.
• Figure 4: $\ell^1$ error for the Logistic equation starting at $x_0 = 0.01$ to give a sigmoid curve with increasing additive Gaussian noise. This utilizes follows the same color scheme as Figure \ref{['fig: lorenz_noise_var']}. We use $s = 3$, $\lambda = 0.05$, and a library of degree 4 or less polynomials.
• Figure 5: $\ell^1$ error for the Logistic Equation with increasing additive Gaussian noise. This has initial condition $x_0 = 10$ resulting in an exponential decay. The library contains polynomials of degree 10 or less. The hyperparameters are $s = 3$ and $\lambda = 0.001$. The right figure is shortened because the error blows up for the SINDy algorithms.