Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ridge Boosting is Both Robust and Efficient

David Bruns-Smith, Zhongming Xie, Avi Feller

Abstract

Estimators in statistics and machine learning must typically trade off between efficiency, having low variance for a fixed target, and distributional robustness, such as multiaccuracy, or having low bias over a range of possible targets. In this paper, we consider a simple estimator, ridge boosting: starting with any initial predictor, perform a single boosting step with (kernel) ridge regression. Surprisingly, we show that ridge boosting simultaneously achieves both efficiency and distributional robustness: for target distribution shifts that lie within an RKHS unit ball, this estimator maintains low bias across all such shifts and has variance at the semiparametric efficiency bound for each target. In addition to bridging otherwise distinct research areas, this result has immediate practical value. Since ridge boosting uses only data from the source distribution, researchers can train a single model to obtain both robust and efficient estimates for multiple target estimands at the same time, eliminating the need to fit separate semiparametric efficient estimators for each target. We assess this approach through simulations and an application estimating the age profile of retirement income.

Ridge Boosting is Both Robust and Efficient

Abstract

Estimators in statistics and machine learning must typically trade off between efficiency, having low variance for a fixed target, and distributional robustness, such as multiaccuracy, or having low bias over a range of possible targets. In this paper, we consider a simple estimator, ridge boosting: starting with any initial predictor, perform a single boosting step with (kernel) ridge regression. Surprisingly, we show that ridge boosting simultaneously achieves both efficiency and distributional robustness: for target distribution shifts that lie within an RKHS unit ball, this estimator maintains low bias across all such shifts and has variance at the semiparametric efficiency bound for each target. In addition to bridging otherwise distinct research areas, this result has immediate practical value. Since ridge boosting uses only data from the source distribution, researchers can train a single model to obtain both robust and efficient estimates for multiple target estimands at the same time, eliminating the need to fit separate semiparametric efficient estimators for each target. We assess this approach through simulations and an application estimating the age profile of retirement income.
Paper Structure (40 sections, 8 theorems, 78 equations, 2 figures)

This paper contains 40 sections, 8 theorems, 78 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Paper organization.
  3. Related literature
  4. Problem setup: Robustness vs efficiency
  5. Notation
  6. Defining our estimation target
  7. Plug-in estimation and regularization bias
  8. Robustness: Constraining worst-case bias across many unknown targets
  9. Efficiency: Unbiased estimate with the smallest variance for a single, known target
  10. Ridge boosting simultaneously achieves robustness and efficiency
  11. Ridge boosting
  12. Setup.
  13. Estimator.
  14. Ridge boosting is multiaccurate
  15. Ridge boosting is semiparametrically efficient
  16. ...and 25 more sections

Key Result

Proposition 1

Let $\Theta$ be some set of functionals $\theta$ such that def:target-estimand and Assumption asm:continuity hold. Let $\mathcal{A}$ be the corresponding set of Riesz representers: Then if $\hat{\gamma}_\text{ma}$ is $(\mathcal{A}, a)$-multiaccurate:

Figures (2)

  • Figure 1: The empirical coverage comparisons between single kernel ridge outcome regression and kernel-ridge-boosted estimator across sample-sizes and the three environments $\mu \in \{0, 1, -1\}$. The blue lines with triangle markers plot the coverage for the base ridge model, and the orange line with circular markers for the once-boosted ridge model. The dotted line is at 0.95.
  • Figure 2: The left panel shows our point estimates for the age profile of income using a naive kernel ridge regression plug-in estimator and our ridge boosting estimator. The dotted lines are the corresponding 95% confidence intervals and the width of the intervals are plotted in the right panel.

Theorems & Definitions (17)

  • Definition 1: Target Estimand
  • Definition 2: Riesz representer
  • Definition 3: Multiaccuracy
  • Proposition 1
  • Remark 1: Distributionally-Robust Optimization
  • Definition 4: Multi-accuracy error
  • Theorem 1
  • Remark 2
  • Proposition 2
  • Remark 3
  • ...and 7 more