Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Systems of ODEs Parameters Estimation by Using Stochastic Newton-Raphson and Gradient Descent Methods

S. Syafiie, Aries Subiantoro, Vivi Andasari, Fernando Tadeo

TL;DR

This work addresses parameter estimation for systems of ordinary differential equations by recasting the problem through a discrete derivative residual and a Taylor-based expansion, enabling direct application of Newton-Raphson and Gradient Descent methods. It extends these methods to stochastic variants (SNR, SGD) to reduce computation on large-scale datasets and compares them against nonlinear least squares across three representative ODE systems: a two-state nonlinear population model, the Lorenz chaotic system, and an activator-inhibitor circuit. Empirical results show that NR achieves rapid, accurate parameter recovery, while GD provides robustness in chaotic regimes albeit with more iterations; in all cases, the proposed methods outperform constrained NLS in both fit quality and error metrics such as bias, MAE, MAPE, RMSE, and $R^2$. The stochastic variants further improve scalability and efficiency for big data without sacrificing accuracy. Overall, the methods offer a flexible and effective toolkit for fitting complex ODE models in biology, chemistry, and engineering where data scale and dynamics are challenging.

Abstract

Ordinary differential equations (ODEs) are widely used to describe the time evolution of natural phenomena across various scientific fields. Estimating the parameters of these systems from data is a challenging task, particularly when dealing with nonlinear and high-dimensional models. In this paper, we propose novel methodologies for parameter estimation in systems of ODEs by using the Newton-Raphson (NR) method and Gradient Descent (GD) method. By leveraging the discrete derivative and Taylor expansion, the problem is formulated in a way that enables the application of both methods, allowing for flexible, efficient solutions. Additionally, we extend these approaches to stochastic versions - Stochastic Newton-Raphson (SNR) and Stochastic Gradient Descent (SGD) - to handle large-scale systems with reduced computational cost. The proposed methods are evaluated by using numerical examples, including both linear and nonlinear parameter models, and compare the results to the well-known Nonlinear Least Squares (NLS) method. While NR converges rapidly to the optimal solution, GD demonstrates robustness in handling chaotic systems, though it may occasionally lead to suboptimal results. Overall, the proposed methods provide improved accuracy in parameter estimation for ODE systems, outperforming NLS in terms of error metrics such as bias, mean absolute error (MAE), mean absolute percentage error (MAPE), root mean square error (RMSE), and coefficient of determination R2. These methods offer a valuable tool for fitting ODE models, particularly in scenarios involving big data and complex dynamics.

Systems of ODEs Parameters Estimation by Using Stochastic Newton-Raphson and Gradient Descent Methods

TL;DR

This work addresses parameter estimation for systems of ordinary differential equations by recasting the problem through a discrete derivative residual and a Taylor-based expansion, enabling direct application of Newton-Raphson and Gradient Descent methods. It extends these methods to stochastic variants (SNR, SGD) to reduce computation on large-scale datasets and compares them against nonlinear least squares across three representative ODE systems: a two-state nonlinear population model, the Lorenz chaotic system, and an activator-inhibitor circuit. Empirical results show that NR achieves rapid, accurate parameter recovery, while GD provides robustness in chaotic regimes albeit with more iterations; in all cases, the proposed methods outperform constrained NLS in both fit quality and error metrics such as bias, MAE, MAPE, RMSE, and . The stochastic variants further improve scalability and efficiency for big data without sacrificing accuracy. Overall, the methods offer a flexible and effective toolkit for fitting complex ODE models in biology, chemistry, and engineering where data scale and dynamics are challenging.

Abstract

Ordinary differential equations (ODEs) are widely used to describe the time evolution of natural phenomena across various scientific fields. Estimating the parameters of these systems from data is a challenging task, particularly when dealing with nonlinear and high-dimensional models. In this paper, we propose novel methodologies for parameter estimation in systems of ODEs by using the Newton-Raphson (NR) method and Gradient Descent (GD) method. By leveraging the discrete derivative and Taylor expansion, the problem is formulated in a way that enables the application of both methods, allowing for flexible, efficient solutions. Additionally, we extend these approaches to stochastic versions - Stochastic Newton-Raphson (SNR) and Stochastic Gradient Descent (SGD) - to handle large-scale systems with reduced computational cost. The proposed methods are evaluated by using numerical examples, including both linear and nonlinear parameter models, and compare the results to the well-known Nonlinear Least Squares (NLS) method. While NR converges rapidly to the optimal solution, GD demonstrates robustness in handling chaotic systems, though it may occasionally lead to suboptimal results. Overall, the proposed methods provide improved accuracy in parameter estimation for ODE systems, outperforming NLS in terms of error metrics such as bias, mean absolute error (MAE), mean absolute percentage error (MAPE), root mean square error (RMSE), and coefficient of determination R2. These methods offer a valuable tool for fitting ODE models, particularly in scenarios involving big data and complex dynamics.
Paper Structure (21 sections, 3 theorems, 67 equations, 6 figures, 10 tables, 4 algorithms)

This paper contains 21 sections, 3 theorems, 67 equations, 6 figures, 10 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Methodology Development
  3. Newton-Raphson Method
  4. Gradient Descent Method
  5. Illustrative Numerical Application
  6. Fitting a population system
  7. Implementation NR method
  8. Implementation GD method
  9. Numerical implementation
  10. Comparison to nonlinear least square
  11. Fitting a Lorenz's system
  12. Implementation NR method
  13. Implementation GD method
  14. Numerical implementation
  15. Comparison to nonlinear LS method
  16. ...and 6 more sections

Key Result

Theorem 1

Let $g_j(t,x,a):=f_j(t,x,a)-\frac{dx}{dt}$ be twice continuously differentiable, by considering that $\nabla_ag_j(t,x,a)\ne 0$, thus by sequence converges to $a^*$ as $i\to\infty$. Then for $i$ sufficient large, where $H_j(t,x,a^*)$ is Hessian matrix of $g(t,x,a^*)$ and $I_j$ is the standard basis of $\mathbb{R}^n$. Thus $a_i$ converges to $a^*$ quadratically.

Figures (6)

  • Figure 1: population dynamic Example: scattered plots are original data, solid lines are estimated data
  • Figure 2: first example NLS fitting: dotted is data, continuous line is approximated data
  • Figure 3: Lorenz's system example: marked plots are original data, solid lines are estimated data
  • Figure 4: Lorenz's system example NLS method: marked plots are original data, solid lines are estimated data by using constrained NLS method
  • Figure 5: Activator-inhibitor example implementation: scatter plots are original data, solid lines are estimated data
  • ...and 1 more figures

Theorems & Definitions (5)

  • Theorem 1
  • proof
  • Corollary 1.1
  • Theorem 2
  • proof