Regularization via Mass Transportation

TL;DR

The paper reframes regularization as distributionally robust optimization by minimizing the worst-case expected loss over a Wasserstein ball around the empirical distribution, yielding a principled, data-driven regularization mechanism. It provides tractable reformulations for linear and nonlinear models, including RKHS kernelization and neural networks, along with generalization bounds derived from Wasserstein concentration. A key insight is that classical regularization emerges as a special case when transport costs in the output space are prohibitive, while the framework delivers test-error guarantees and worst-case-distribution analysis. Empirical results on MNIST and PASCAL VOC demonstrate improved out-of-sample performance and practical learnability of regularization parameters, underscoring the approach’s utility for robust prediction under distributional uncertainty.

Abstract

The goal of regression and classification methods in supervised learning is to minimize the empirical risk, that is, the expectation of some loss function quantifying the prediction error under the empirical distribution. When facing scarce training data, overfitting is typically mitigated by adding regularization terms to the objective that penalize hypothesis complexity. In this paper we introduce new regularization techniques using ideas from distributionally robust optimization, and we give new probabilistic interpretations to existing techniques. Specifically, we propose to minimize the worst-case expected loss, where the worst case is taken over the ball of all (continuous or discrete) distributions that have a bounded transportation distance from the (discrete) empirical distribution. By choosing the radius of this ball judiciously, we can guarantee that the worst-case expected loss provides an upper confidence bound on the loss on test data, thus offering new generalization bounds. We prove that the resulting regularized learning problems are tractable and can be tractably kernelized for many popular loss functions. We validate our theoretical out-of-sample guarantees through simulated and empirical experiments.

Figures (3)

• Figure 1: Average out-of-sample CCR scores of the DRSVM with pre-selected parameters.
• Figure 2: Results of the threenorm classification problem.
• Figure 3: Discretization points (input images) of the worst-case distribution for different $\rho$ and $\kappa$. Red frames indicate that the corresponding labels are flipped under the worst-case distribution.

Theorems & Definitions (74)

• Remark 2.1: Convex approximation
• Definition 2.2: Wasserstein metric
• Remark 2.3: Support information
• Theorem 3.1: Distributionally robust linear regression
• Corollary 3.2: Robust regression
• Corollary 3.3: Support vector regression
• Corollary 3.4: Quantile regression
• Remark 3.5: Relation to classical regularization
• Theorem 3.6: Extremal distributions in linear regression
• Remark 3.8: Minimal dispersion