Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An L-BFGS-B approach for linear and nonlinear system identification under $\ell_1$ and group-Lasso regularization

Alberto Bemporad

TL;DR

The paper addresses robust identification of linear and nonlinear discrete-time state-space systems under sparsity-promoting regularization. It introduces a bound-constrained L-BFGS-B-based framework capable of handling $\\ell_1$ and group-Lasso penalties and non-quadratic losses, including RNN-based dynamics. A key contribution is a split-variable reformulation that enables efficient gradient-based optimization on a condensed loss that eliminates hidden states, together with stability and DC-gain constraints implemented as penalties or bounds. Empirically, the method delivers competitive performance on linear benchmarks and demonstrates practical nonlinear SYSID for a multi-input/multi-output industrial robot, supported by an accessible Python implementation in jax-sysid.

Abstract

In this paper, we propose a very efficient numerical method based on the L-BFGS-B algorithm for identifying linear and nonlinear discrete-time state-space models, possibly under $\ell_1$ and group-Lasso regularization for reducing model complexity. For the identification of linear models, we show that, compared to classical linear subspace methods, the approach often provides better results, is much more general in terms of the loss and regularization terms used (such as penalties for enforcing system stability), and is also more stable from a numerical point of view. The proposed method not only enriches the existing set of linear system identification tools but can also be applied to identifying a very broad class of parametric nonlinear state-space models, including recurrent neural networks. We illustrate the approach on synthetic and experimental datasets and apply it to solve a challenging industrial robot benchmark for nonlinear multi-input/multi-output system identification. A Python implementation of the proposed identification method is available in the package jax-sysid, available at https://github.com/bemporad/jax-sysid.

An L-BFGS-B approach for linear and nonlinear system identification under $\ell_1$ and group-Lasso regularization

TL;DR

The paper addresses robust identification of linear and nonlinear discrete-time state-space systems under sparsity-promoting regularization. It introduces a bound-constrained L-BFGS-B-based framework capable of handling and group-Lasso penalties and non-quadratic losses, including RNN-based dynamics. A key contribution is a split-variable reformulation that enables efficient gradient-based optimization on a condensed loss that eliminates hidden states, together with stability and DC-gain constraints implemented as penalties or bounds. Empirically, the method delivers competitive performance on linear benchmarks and demonstrates practical nonlinear SYSID for a multi-input/multi-output industrial robot, supported by an accessible Python implementation in jax-sysid.

Abstract

In this paper, we propose a very efficient numerical method based on the L-BFGS-B algorithm for identifying linear and nonlinear discrete-time state-space models, possibly under and group-Lasso regularization for reducing model complexity. For the identification of linear models, we show that, compared to classical linear subspace methods, the approach often provides better results, is much more general in terms of the loss and regularization terms used (such as penalties for enforcing system stability), and is also more stable from a numerical point of view. The proposed method not only enriches the existing set of linear system identification tools but can also be applied to identifying a very broad class of parametric nonlinear state-space models, including recurrent neural networks. We illustrate the approach on synthetic and experimental datasets and apply it to solve a challenging industrial robot benchmark for nonlinear multi-input/multi-output system identification. A Python implementation of the proposed identification method is available in the package jax-sysid, available at https://github.com/bemporad/jax-sysid.
Paper Structure (29 sections, 5 theorems, 29 equations, 4 figures, 3 tables, 1 algorithm)

This paper contains 29 sections, 5 theorems, 29 equations, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. System identification problem
  3. Special cases
  4. Linear state-space models
  5. Training recurrent neural networks
  6. $\ell_2$ and $\ell_1$ regularization
  7. Group-Lasso regularization
  8. Handling multiple experiments
  9. Stability constraints
  10. DC gain
  11. Non-smooth nonlinear optimization
  12. Constraints on model parameters and initial states
  13. Preventing numerical overflows
  14. Initial state reconstruction for validation
  15. Numerical results
  16. ...and 14 more sections

Key Result

Lemma 1

Let $\Sigma\triangleq (A,B,C,D)$ be an asymptotically stable discrete-time linear system. Then there exists a transformation $T$ such that the equivalent system $\bar{\Sigma}\triangleq (\bar{A},\bar{B},\bar{C},\bar{D})$ is such that $\|\bar{A}\|_2<1$, where $\bar{A}=TAT^{-1}$, $\bar{B}=TB$, $\bar{C}

Figures (4)

  • Figure 1: $R^2$-score on training data and resulting model order obtained with the group-Lasso penalty \ref{['eq:group-Lasso-x']} for different values of $\tau_g$.
  • Figure 2: $R^2$-score on training data and resulting number of model inputs under the group-Lasso penalty \ref{['eq:group-Lasso-u']} for different values of $\tau_g$.
  • Figure 3: Average $R^2$-score of open-loop predictions on training/test data and model sparsity, as a function of $\tau$. For each value of $\tau$, we show the best $\overline{R^2}$-score obtained on test data out of 30 runs and the corresponding $\overline{R^2}$-score on training data and model sparsity.
  • Figure 4: Average R$^2$-score of open-loop predictions on test data from state estimates $\hat{x}_{k|k}$ obtained by EKF ($\tau$=0.008).

Theorems & Definitions (5)

  • Lemma 1
  • Lemma 2
  • Corollary 1
  • Lemma 3
  • Corollary 2