Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automatic Outlier Rectification via Optimal Transport

Jose Blanchet, Jiajin Li, Markus Pelger, Greg Zanotti

TL;DR

The paper tackles outlier contamination by proposing a joint rectification-estimation framework based on optimal transport with a concave cost, forming a rectification set \(\mathcal{R}(\mathbb{P}'_n)\) around the contaminated distribution and selecting the best rectified distribution for estimation. It develops strong duality results that turn the infinite-dimensional inner problem into finite convex programs and reveals connections to adaptive quantile regression for mean estimation and LAD regression. Empirically, the method improves mean estimation, LAD regression, and option-implied volatility surface estimation, delivering smoother surfaces and lower error than conventional baselines, with cross-validated budget \(\delta\) guiding outlier rectification. The approach demonstrates robust performance in both kernelized IVS and Dupire neural-network contexts, suggesting broad applicability of automatic outlier rectification in statistical learning under contamination.

Abstract

In this paper, we propose a novel conceptual framework to detect outliers using optimal transport with a concave cost function. Conventional outlier detection approaches typically use a two-stage procedure: first, outliers are detected and removed, and then estimation is performed on the cleaned data. However, this approach does not inform outlier removal with the estimation task, leaving room for improvement. To address this limitation, we propose an automatic outlier rectification mechanism that integrates rectification and estimation within a joint optimization framework. We take the first step to utilize the optimal transport distance with a concave cost function to construct a rectification set in the space of probability distributions. Then, we select the best distribution within the rectification set to perform the estimation task. Notably, the concave cost function we introduced in this paper is the key to making our estimator effectively identify the outlier during the optimization process. We demonstrate the effectiveness of our approach over conventional approaches in simulations and empirical analyses for mean estimation, least absolute regression, and the fitting of option implied volatility surfaces.

Automatic Outlier Rectification via Optimal Transport

TL;DR

The paper tackles outlier contamination by proposing a joint rectification-estimation framework based on optimal transport with a concave cost, forming a rectification set \(\mathcal{R}(\mathbb{P}'_n)\) around the contaminated distribution and selecting the best rectified distribution for estimation. It develops strong duality results that turn the infinite-dimensional inner problem into finite convex programs and reveals connections to adaptive quantile regression for mean estimation and LAD regression. Empirically, the method improves mean estimation, LAD regression, and option-implied volatility surface estimation, delivering smoother surfaces and lower error than conventional baselines, with cross-validated budget guiding outlier rectification. The approach demonstrates robust performance in both kernelized IVS and Dupire neural-network contexts, suggesting broad applicability of automatic outlier rectification in statistical learning under contamination.

Abstract

In this paper, we propose a novel conceptual framework to detect outliers using optimal transport with a concave cost function. Conventional outlier detection approaches typically use a two-stage procedure: first, outliers are detected and removed, and then estimation is performed on the cleaned data. However, this approach does not inform outlier removal with the estimation task, leaving room for improvement. To address this limitation, we propose an automatic outlier rectification mechanism that integrates rectification and estimation within a joint optimization framework. We take the first step to utilize the optimal transport distance with a concave cost function to construct a rectification set in the space of probability distributions. Then, we select the best distribution within the rectification set to perform the estimation task. Notably, the concave cost function we introduced in this paper is the key to making our estimator effectively identify the outlier during the optimization process. We demonstrate the effectiveness of our approach over conventional approaches in simulations and empirical analyses for mean estimation, least absolute regression, and the fitting of option implied volatility surfaces.
Paper Structure (53 sections, 6 theorems, 38 equations, 16 figures, 5 tables, 2 algorithms)

This paper contains 53 sections, 6 theorems, 38 equations, 16 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. DRO and Robust Statistics as Adversarial Games
  3. Automatic Outlier Rectification Mechanism
  4. Reformulation Results
  5. Mean Estimation
  6. Least Absolute Deviation (LAD) Regression and More General Forms
  7. Computational Procedure
  8. Experimental Results
  9. Mean Estimation and LAD Regression Experiments
  10. Options Volatility Experiments
  11. Kernelized Volatility Surface Estimation
  12. Data.
  13. Benchmarks.
  14. Kernelized Regression Problem.
  15. Results.
  16. ...and 38 more sections

Key Result

Proposition 1

Suppose that $\ell(\theta,\cdot)$ is lower semicontinuous and integrable under $\mathbb{P}_n'$ for any $\theta \in \Theta$. Then, the strong duality holds, i.e.,

Figures (16)

  • Figure 1: Data-Driven Decision Making Cycle
  • Figure 2: The rectified data generated by our estimator with concave cost function ($r=0.5)$.
  • Figure 3: The rectified data generated by our estimator with convex cost function ($r=2)$.
  • Figure 4: Visualization
  • Figure 5: Irregular objective function.
  • ...and 11 more figures

Theorems & Definitions (19)

  • Definition 3.1: Rectification Set
  • Definition 3.2
  • Proposition 1: Strong Duality
  • Theorem 2: Mean Estimation
  • Remark 4.1
  • Proposition 3: Characterization of Rectified Distribution
  • Remark 4.2
  • Theorem 4: LAD Regression
  • Remark 5.1
  • Remark B.1
  • ...and 9 more