Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distribution-Free Confidence Ellipsoids for Ridge Regression with PAC Bounds

Szabolcs Szentpéteri, Balázs Csanád Csáji

TL;DR

This work introduces a ridge-regularized extension of Sign-Perturbed Sums (SPS) ellipsoidal outer approximations (EOA) to produce non-asymptotic, distribution-free confidence regions for linear models. The proposed RR-SPS framework includes an Indicator test and an EOA construction centered at the ridge estimate hat theta, with a Mahalanobis metric using bar R to reflect regularization. The authors derive PAC upper bounds on the radius of the RR-SPS EOA, explicitly characterizing how the regularization parameter and excitation properties influence region size, and they prove concentration results for key matrices to tighten these bounds. Simulation studies confirm the qualitative effects of regularization on region size and show alignment between empirical region sizes and the theoretical PAC bounds, validating the approach for finite-sample uncertainty quantification in ridge regression.

Abstract

Linearly parametrized models are widely used in control and signal processing, with the least-squares (LS) estimate being the archetypical solution. When the input is insufficiently exciting, the LS problem may be unsolvable or numerically unstable. This issue can be resolved through regularization, typically with ridge regression. Although regularized estimators reduce the variance error, it remains important to quantify their estimation uncertainty. A possible approach for linear regression is to construct confidence ellipsoids with the Sign-Perturbed Sums (SPS) ellipsoidal outer approximation (EOA) algorithm. The SPS EOA builds non-asymptotic confidence ellipsoids under the assumption that the noises are independent and symmetric about zero. This paper introduces an extension of the SPS EOA algorithm to ridge regression, and derives probably approximately correct (PAC) upper bounds for the resulting region sizes. Compared with previous analyses, our result explicitly show how the regularization parameter affects the region sizes, and provide tighter bounds under weaker excitation assumptions. Finally, the practical effect of regularization is also demonstrated via simulation experiments.

Distribution-Free Confidence Ellipsoids for Ridge Regression with PAC Bounds

TL;DR

This work introduces a ridge-regularized extension of Sign-Perturbed Sums (SPS) ellipsoidal outer approximations (EOA) to produce non-asymptotic, distribution-free confidence regions for linear models. The proposed RR-SPS framework includes an Indicator test and an EOA construction centered at the ridge estimate hat theta, with a Mahalanobis metric using bar R to reflect regularization. The authors derive PAC upper bounds on the radius of the RR-SPS EOA, explicitly characterizing how the regularization parameter and excitation properties influence region size, and they prove concentration results for key matrices to tighten these bounds. Simulation studies confirm the qualitative effects of regularization on region size and show alignment between empirical region sizes and the theoretical PAC bounds, validating the approach for finite-sample uncertainty quantification in ridge regression.

Abstract

Linearly parametrized models are widely used in control and signal processing, with the least-squares (LS) estimate being the archetypical solution. When the input is insufficiently exciting, the LS problem may be unsolvable or numerically unstable. This issue can be resolved through regularization, typically with ridge regression. Although regularized estimators reduce the variance error, it remains important to quantify their estimation uncertainty. A possible approach for linear regression is to construct confidence ellipsoids with the Sign-Perturbed Sums (SPS) ellipsoidal outer approximation (EOA) algorithm. The SPS EOA builds non-asymptotic confidence ellipsoids under the assumption that the noises are independent and symmetric about zero. This paper introduces an extension of the SPS EOA algorithm to ridge regression, and derives probably approximately correct (PAC) upper bounds for the resulting region sizes. Compared with previous analyses, our result explicitly show how the regularization parameter affects the region sizes, and provide tighter bounds under weaker excitation assumptions. Finally, the practical effect of regularization is also demonstrated via simulation experiments.
Paper Structure (12 sections, 55 equations, 1 figure, 4 tables)

This paper contains 12 sections, 55 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Problem setting
  3. Assumptions
  4. The ridge regression SPS algorithm
  5. The ridge regression SPS-Indicator algorithm
  6. The ridge regression SPS EOA
  7. Sample complexity of RR-SPS EOA
  8. Proof of Lemma \ref{['lem:tilde_K_concentration']}
  9. Proof of Lemma \ref{['lemma:K_concentration']}
  10. Proof of Lemma \ref{['lemma:proj_indicator']}
  11. Simulation Experiments
  12. Conclusion

Figures (1)

  • Figure 1: Sizes of $0.9$-level RR-SPS EOA regions with different regularization parameter values $\lambda$ for $n=250$.