Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Empirical Bayes approach to ARX Estimation

Timofei Leahu, Giorgio Picci

TL;DR

This paper analyzes Empirical Bayes approaches for ARX time-series, contrasting marginal Bayes with full Empirical Bayes under finite data. It develops a backward conditional Kalman filter to sequentially estimate hyperparameters (mean μ, prior covariance Π, and noise variance σ^2) and provides explicit MSE expressions to compare the two estimators. Theoretical results show that marginal Bayes can outperform EB for small prior variance and limited data, while EB can dominate when the prior is large or data are highly informative; simulations with fixed and slowly varying parameters corroborate these findings. The work highlights non-asymptotic behavior and offers practical guidance for selecting EB vs marginal Bayes in finite-sample ARX identification.

Abstract

Empirical Bayes inference is based on estimation of the parameters of an a priori distribution from the observed data. The estimation technique of the parameters of the prior, called hyperparameters, is based on the marginal distribution obtained by integrating the joint density of the model with respect to the prior. This is a key step which needs to be properly adapted to the problem at hand. In this paper we study Empirical Bayes inference of linear autoregressive models with inputs (ARX models) for time series and compare the performance of the marginal parametric estimator with that a full Empirical Bayesian analysis based on the estimated prior. Such a comparison, can only make sense for a (realistic) finite data length. In this setting, we propose a new estimation technique of the hyperparameters by a sequential Bayes procedure which is essentially a backward Kalman filter. It turns out that for finite data length the marginal Bayes tends to behave slightly better than the full Empirical Bayesian parameter estimator and so also in the case of slowly varying random parameters.

An Empirical Bayes approach to ARX Estimation

TL;DR

This paper analyzes Empirical Bayes approaches for ARX time-series, contrasting marginal Bayes with full Empirical Bayes under finite data. It develops a backward conditional Kalman filter to sequentially estimate hyperparameters (mean μ, prior covariance Π, and noise variance σ^2) and provides explicit MSE expressions to compare the two estimators. Theoretical results show that marginal Bayes can outperform EB for small prior variance and limited data, while EB can dominate when the prior is large or data are highly informative; simulations with fixed and slowly varying parameters corroborate these findings. The work highlights non-asymptotic behavior and offers practical guidance for selecting EB vs marginal Bayes in finite-sample ARX identification.

Abstract

Empirical Bayes inference is based on estimation of the parameters of an a priori distribution from the observed data. The estimation technique of the parameters of the prior, called hyperparameters, is based on the marginal distribution obtained by integrating the joint density of the model with respect to the prior. This is a key step which needs to be properly adapted to the problem at hand. In this paper we study Empirical Bayes inference of linear autoregressive models with inputs (ARX models) for time series and compare the performance of the marginal parametric estimator with that a full Empirical Bayesian analysis based on the estimated prior. Such a comparison, can only make sense for a (realistic) finite data length. In this setting, we propose a new estimation technique of the hyperparameters by a sequential Bayes procedure which is essentially a backward Kalman filter. It turns out that for finite data length the marginal Bayes tends to behave slightly better than the full Empirical Bayesian parameter estimator and so also in the case of slowly varying random parameters.
Paper Structure (9 sections, 75 equations, 3 figures, 3 tables)

This paper contains 9 sections, 75 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. ARX models and pseudo-linear regression
  3. Preliminaries on Conditionally Gaussian Linear Models and Marginals
  4. Relations between the marginal and Empirical Bayes Estimates
  5. The MSE of the Empirical Bayes estimators
  6. Hyperparameter estimation
  7. A Comparison Example
  8. Conclusions
  9. The Backward Kalman filter for Conditionally Gaussian ARX models

Figures (3)

  • Figure 1: Behavior of the two variances and the bias norm squared (in red).
  • Figure 2: Comparison between Marginal and Empirical Bayes MSE.
  • Figure 3: Outputs corresponding to N=100 and $\lambda=0.01, 0.02$ for variable parameters. The plots with fixed parameters in the two graphics are identical.

Theorems & Definitions (1)

  • Example 1