Bayes and Biased Estimators Without Hyper-parameter Estimation: Comparable Performance to the Empirical-Bayes-Based Regularized Estimator

TL;DR

This work designs hyper-parameter-free estimators for ridge-like regularized system identification that match the empirical Bayes (EB) based regularized estimator in an asymptotic XMSE sense. By leveraging XMSE expressions, the authors construct a family of generalized Bayes estimators with the prior $\pi(\theta)=\|\theta\|_2^{2-n}(C_1\|\theta\|_2+C_2\|\theta\|_2^{-1})^2$ and a closed-form biased estimator, both achieving $XMSE$ equal to that of the EB-based regularized estimator while avoiding hyper-parameter estimation. Numerical simulations show comparable performance to EB-based regularization with substantially improved computational efficiency, especially for the biased estimator, across varying problem sizes. The approach provides a practical pathway to hyper-parameter-free, regularized estimators in linear regression for system identification and motivates extensions to broader kernel families and kernels beyond identity Gram matrix settings.

Abstract

Regularized system identification has become a significant complement to more classical system identification. It has been numerically shown that kernel-based regularized estimators often perform better than the maximum likelihood estimator in terms of minimizing mean squared error (MSE). However, regularized estimators often require hyper-parameter estimation. This paper focuses on ridge regression and the regularized estimator by employing the empirical Bayes hyper-parameter estimator. We utilize the excess MSE to quantify the MSE difference between the empirical-Bayes-based regularized estimator and the maximum likelihood estimator for large sample sizes. We then exploit the excess MSE expressions to develop both a family of generalized Bayes estimators and a family of closed-form biased estimators. They have the same excess MSE as the empirical-Bayes-based regularized estimator but eliminate the need for hyper-parameter estimation. Moreover, we conduct numerical simulations to show that the performance of these new estimators is comparable to the empirical-Bayes-based regularized estimator, while computationally, they are more efficient.

Figures (3)

• Figure 1: Sample means of the sample MSE and the average FIT of $\hat{\bm\theta}^{\mathop{\mathrm{ML}}\limits}$, $\hat{\bm\theta}^{\text{R}}(\hat{\eta}_{\mathop{\mathrm{EB}}\limits})$, $\hat{\bm\theta}^{\mathop{\mathrm{Bayes}}\limits,\mathop{\mathrm{EB}}\limits}$ and $\hat{\bm\theta}^{\mathop{\mathrm{Biased}}\limits,\mathop{\mathrm{EB}}\limits}$ for $n=1$ and $N=5$.
• Figure 2: Sample means of the sample MSE and the average FIT of $\hat{\bm\theta}^{\mathop{\mathrm{ML}}\limits}$, $\hat{\bm\theta}^{\text{R}}(\hat{\eta}_{\mathop{\mathrm{EB}}\limits})$, $\hat{\bm\theta}^{\mathop{\mathrm{Bayes}}\limits,\mathop{\mathrm{EB}}\limits}$ and $\hat{\bm\theta}^{\mathop{\mathrm{Biased}}\limits,\mathop{\mathrm{EB}}\limits}$ for $n=5$ and $N=15$.
• Figure 3: Sample means of the sample MSE and the average FIT of $\hat{\bm\theta}^{\mathop{\mathrm{ML}}\limits}$, $\hat{\bm\theta}^{\text{R}}(\hat{\eta}_{\mathop{\mathrm{EB}}\limits})$, $\hat{\bm\theta}^{\mathop{\mathrm{Bayes}}\limits,\mathop{\mathrm{EB}}\limits}$ and $\hat{\bm\theta}^{\mathop{\mathrm{Biased}}\limits,\mathop{\mathrm{EB}}\limits}$ for $n=80$ and $N=360$.