Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Wasserstein Profile Inference and Applications to Machine Learning

Jose Blanchet, Yang Kang, Karthyek Murthy

TL;DR

<p>We address regularization in machine learning by casting popular estimators as distributionally robust optimization problems with Wasserstein-based uncertainty sets. The authors introduce Robust Wasserstein Profile Inference (RWPI), a novel inference framework that uses transport-cost-based profile functions to automatically calibrate the uncertainty radius and, hence, regularization parameters without cross-validation. They prove dual representations and asymptotic limit theorems for the Robust Wasserstein Profile (RWP) function, enabling confidence regions and data-driven parameter tuning for square-root LASSO and regularized logistic regression, including high-dimensional settings. Numerical experiments corroborate the theoretical results, showing competitive performance with cross-validation while avoiding repetitive tuning, and illustrate practical guidance for selecting regularization in DRO-based ML estimators.</p>

Abstract

We show that several machine learning estimators, including square-root LASSO (Least Absolute Shrinkage and Selection) and regularized logistic regression can be represented as solutions to distributionally robust optimization (DRO) problems. The associated uncertainty regions are based on suitably defined Wasserstein distances. Hence, our representations allow us to view regularization as a result of introducing an artificial adversary that perturbs the empirical distribution to account for out-of-sample effects in loss estimation. In addition, we introduce RWPI (Robust Wasserstein Profile Inference), a novel inference methodology which extends the use of methods inspired by Empirical Likelihood to the setting of optimal transport costs (of which Wasserstein distances are a particular case). We use RWPI to show how to optimally select the size of uncertainty regions, and as a consequence, we are able to choose regularization parameters for these machine learning estimators without the use of cross validation. Numerical experiments are also given to validate our theoretical findings.

Robust Wasserstein Profile Inference and Applications to Machine Learning

TL;DR

<p>We address regularization in machine learning by casting popular estimators as distributionally robust optimization problems with Wasserstein-based uncertainty sets. The authors introduce Robust Wasserstein Profile Inference (RWPI), a novel inference framework that uses transport-cost-based profile functions to automatically calibrate the uncertainty radius and, hence, regularization parameters without cross-validation. They prove dual representations and asymptotic limit theorems for the Robust Wasserstein Profile (RWP) function, enabling confidence regions and data-driven parameter tuning for square-root LASSO and regularized logistic regression, including high-dimensional settings. Numerical experiments corroborate the theoretical results, showing competitive performance with cross-validation while avoiding repetitive tuning, and illustrate practical guidance for selecting regularization in DRO-based ML estimators.</p>

Abstract

We show that several machine learning estimators, including square-root LASSO (Least Absolute Shrinkage and Selection) and regularized logistic regression can be represented as solutions to distributionally robust optimization (DRO) problems. The associated uncertainty regions are based on suitably defined Wasserstein distances. Hence, our representations allow us to view regularization as a result of introducing an artificial adversary that perturbs the empirical distribution to account for out-of-sample effects in loss estimation. In addition, we introduce RWPI (Robust Wasserstein Profile Inference), a novel inference methodology which extends the use of methods inspired by Empirical Likelihood to the setting of optimal transport costs (of which Wasserstein distances are a particular case). We use RWPI to show how to optimally select the size of uncertainty regions, and as a consequence, we are able to choose regularization parameters for these machine learning estimators without the use of cross validation. Numerical experiments are also given to validate our theoretical findings.

Paper Structure

This paper contains 39 sections, 20 theorems, 233 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Regularization in Linear Regression
  3. DRO representation of square-root LASSO.
  4. Optimal choice of the radius $\delta$.
  5. The associated Wasserstein Profile Function.
  6. Further intuition behind the RWP function.
  7. A broader perspective of our contribution
  8. On related literature in Robust Optimization, Distributionally Robust Optimization and Optimal Transport
  9. Connections to related inference literature
  10. Organization of the paper
  11. Optimal Transport Definitions and DRO Representations of Machine Learning Estimators
  12. Optimal Transport Costs and Wasserstein Distances
  13. DRO formulation using optimal transport costs
  14. Dual form of the DRO formulation (\ref{['RWPI-DRO']})
  15. Distributionally Robust Representations
  16. ...and 24 more sections

Key Result

Proposition 1

Let $c: \mathbb{R}^{d+1} \times \mathbb{R}^{d+1} \rightarrow [0,\infty]$ be a lower semi-continuous cost function satisfying $c((x,y),(x^\prime,y^\prime)) = 0$ whenever $(x,y) = (x^\prime,y^\prime).$ For $\gamma\geq0$ and loss functions $l(x,y;\beta)$ that are upper semi-continuous in $(x,y)$ for ea Then

Figures (1)

  • Figure 1: Illustration of RWP function evaluated at $\beta_{\ast}$

Theorems & Definitions (41)

  • Example 1: Square-root Lasso
  • Example 2: Regularized Logistic Regression
  • Proposition 1
  • Proposition 2: DR linear regression with square loss
  • Theorem 1
  • Theorem 2: Regularized regression for Classification
  • Proposition 3
  • Example 3
  • Theorem 3
  • Proposition 4
  • ...and 31 more