Information theory and discriminative sampling for model discovery

TL;DR

This work integrates Fisher information and entropy-based metrics into data-driven model discovery with SINDy to quantify how informatively different data segments contribute to learning. It derives a rigorous FIM-based framework, analyzes spectral properties, and demonstrates that discriminative sampling and bagging can substantially improve identification accuracy with less data. The authors show that information-guided data collection, active control, and entropy-search strategies yield faster convergence across single-trajectory and multi-trajectory scenarios, including chaotic systems. Practical implications include enhanced sampling design, improved robustness under noise, and a roadmap for integrating information theory with adaptive sensing in dynamical-system identification.

Abstract

Fisher information and Shannon entropy are fundamental tools for understanding and analyzing dynamical systems from complementary perspectives. They can characterize unknown parameters by quantifying the information contained in variables, or measure how different initial trajectories or temporal segments of a trajectory contribute to learning or inferring system dynamics. In this work, we leverage the Fisher Information Matrix (FIM) within the data-driven framework of {\em sparse identification of nonlinear dynamics} (SINDy). We visualize information patterns in chaotic and non-chaotic systems for both single trajectories and multiple initial conditions, demonstrating how information-based analysis can improve sampling efficiency and enhance model performance by prioritizing more informative data. The benefits of statistical bagging are further elucidated through spectral analysis of the FIM. We also illustrate how Fisher information and entropy metrics can promote data efficiency in three scenarios: when only a single trajectory is available, when a tunable control parameter exists, and when multiple trajectories can be freely initialized. As data-driven model discovery continues to gain prominence, principled sampling strategies guided by quantifiable information metrics offer a powerful approach for improving learning efficiency and reducing data requirements.

Figures (14)

• Figure 1: Panel (a) illustrates the Rössler system converging to a centrally located fixed point under the parameter setting $a<0$, $b=0.2$, $c=5.7$. Panel (b) depicts the Van der Pol oscillator exhibiting a stable limit cycle for $\mu = 0.8$.
• Figure 2: The largest eigenvalues $\lambda_1$ of the Fisher Information Matrix (FIM) are shown alongside model simulations for the chaotic Lorenz system. Panel (a) corresponds to a set of initial conditions farther from the orbit, while the panel (b) corresponds to initial condition closer to the orbit.
• Figure 3: The plots are defined on two-dimensional grids of initial conditions spanning different $(x, y)$ combinations, with distinct choices of the $z$-value across panels. Panel (a) presents the $L_2$ coefficient error (loss) of the Lorenz system obtained from training datasets of equal length, illustrating how reconstruction accuracy varies with the informativeness of the initial conditions. Panel (b) shows the number of extreme values encountered in training stage using data of the same length, serving as a proxy for local chaotic sensitivity and numerical stiffness. The $z$-value selections in panel (b) correspond to those used in the respective panels of (a), enabling a direct comparison between sampling discriminability, chaotic sensitivity, and model identification performance.
• Figure 4: The plots are defined on two-dimensional grids of initial conditions spanning different $(x, y)$ combinations, with distinct choices of the $z$-value across panels. The resulting information patterns of the Lorenz system under varying initial conditions are visualized using the largest eigenvalue and spectral skewness of the Fisher Information Matrix (FIM) as quantitative metrics. These measures characterize how informative each initial condition is for model identification, linking regions of higher information score to enhanced discriminative sampling capability and increased sensitivity to the underlying chaotic dynamics.
• Figure 5: The plots are defined on two-dimensional grids of initial conditions spanning different $(x, y)$ combinations, with distinct choices of the $z$-value across panels. Panel (a) presents the $L_2$ coefficient error (loss) of the Rossler attractor obtained from training datasets of equal length; Panel (b) shows the number of extreme values encountered in training stage using data of the same length; Panel (c) presents information patterns of the Rossler attractor under varying initial conditions, using same metrics as in Fig \ref{['fig:lor_3']}.

Theorems & Definitions (19)

• Proposition 3.1: Fisher Information Matrix for Linear Regression
• proof
• Remark 1
• Definition 3.1: Directional Information
• Proposition 3.2: Extremal Information Directions
• Definition 3.2: FIM-Based Information Metrics
• Proposition 3.3: Connection to Estimator Variance
• Theorem 3.1: Additivity of Fisher Information
• proof
• Theorem 3.2: Eigenvalue Bounds for Aggregate FIM