Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Inexact Halpern Iteration with Application to Distributionally Robust Optimization

Ling Liang, Zusen Xu, Kim-Chuan Toh, Jia-Jie Zhu

TL;DR

This work analyzes two inexact variants of the Halpern fixed-point iteration for monotone inclusion problems, establishing $O(k^{-1})$ convergence in the (expected) residue norm under relaxed inexactness tolerances in both deterministic and stochastic settings. It develops a stochastic Halpern scheme using PAGE to handle finite-sum structures, maintaining the same rate guarantees in expectation with controllable variance, and provides explicit rate- and sample-complexity results for different tolerance schemes. The framework is then applied to data-driven Wasserstein distributionally robust optimization (WDRO), reformulating WDRO problems with convex-concave min-max structures as monotone inclusions and solving them inexactly or stochastically. Two WDRO classes are studied: WDRO with Wasserstein ambiguity in supervised learning and WDRO with general convex-concave losses, including practical projections and variance-reduction techniques; numerical experiments validate the approach and demonstrate favorable performance and scalability relative to exact Halpern and PGDA baselines. Overall, the paper extends Halpern-type methods to inexact/stochastic regimes and broad WDRO settings, offering theoretically solid, scalable tools for robust, data-driven optimization.

Abstract

The Halpern iteration for solving monotone inclusion problems has gained increasing interests in recent years due to its simple form and appealing convergence properties. In this paper, we investigate the inexact variants of the scheme in both deterministic and stochastic settings. We conduct extensive convergence analysis and show that by choosing the inexactness tolerances appropriately, the inexact schemes admit an $O(k^{-1})$ convergence rate in terms of the (expected) residue norm. Our results relax the state-of-the-art inexactness conditions employed in the literature while sharing the same competitive convergence properties. We then demonstrate how the proposed methods can be applied for solving two classes of data-driven Wasserstein distributionally robust optimization problems that admit convex-concave min-max optimization reformulations. We highlight its capability of performing inexact computations for distributionally robust learning with stochastic first-order methods and for general nonlinear convex-concave loss functions, which are competitive in the literature.

An Inexact Halpern Iteration with Application to Distributionally Robust Optimization

TL;DR

This work analyzes two inexact variants of the Halpern fixed-point iteration for monotone inclusion problems, establishing convergence in the (expected) residue norm under relaxed inexactness tolerances in both deterministic and stochastic settings. It develops a stochastic Halpern scheme using PAGE to handle finite-sum structures, maintaining the same rate guarantees in expectation with controllable variance, and provides explicit rate- and sample-complexity results for different tolerance schemes. The framework is then applied to data-driven Wasserstein distributionally robust optimization (WDRO), reformulating WDRO problems with convex-concave min-max structures as monotone inclusions and solving them inexactly or stochastically. Two WDRO classes are studied: WDRO with Wasserstein ambiguity in supervised learning and WDRO with general convex-concave losses, including practical projections and variance-reduction techniques; numerical experiments validate the approach and demonstrate favorable performance and scalability relative to exact Halpern and PGDA baselines. Overall, the paper extends Halpern-type methods to inexact/stochastic regimes and broad WDRO settings, offering theoretically solid, scalable tools for robust, data-driven optimization.

Abstract

The Halpern iteration for solving monotone inclusion problems has gained increasing interests in recent years due to its simple form and appealing convergence properties. In this paper, we investigate the inexact variants of the scheme in both deterministic and stochastic settings. We conduct extensive convergence analysis and show that by choosing the inexactness tolerances appropriately, the inexact schemes admit an convergence rate in terms of the (expected) residue norm. Our results relax the state-of-the-art inexactness conditions employed in the literature while sharing the same competitive convergence properties. We then demonstrate how the proposed methods can be applied for solving two classes of data-driven Wasserstein distributionally robust optimization problems that admit convex-concave min-max optimization reformulations. We highlight its capability of performing inexact computations for distributionally robust learning with stochastic first-order methods and for general nonlinear convex-concave loss functions, which are competitive in the literature.
Paper Structure (18 sections, 17 theorems, 114 equations, 2 figures, 1 table)

This paper contains 18 sections, 17 theorems, 114 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Notation and Definitions
  3. An inexact Halpern fixed-point iterative scheme
  4. A stochastic Halpern iteration with PAGE
  5. Data-driven Wasserstein distributionally robust optimization
  6. Wasserstein distributionally robust supervised learning
  7. WDRO with convex-concave loss functions
  8. Numerical experiments
  9. 2-Wasserstein DRO problems with quadratic-linear loss
  10. 2-Wasserstein DRO problems with nonlinear convex-concave loss
  11. Conclusions
  12. Technical lemmas and proofs.
  13. Proofs of Main Results
  14. Proof of Theorem \ref{['thm-convergence-rate']}
  15. Proof of Theorem \ref{['thm-rate-ishi']}
  16. ...and 3 more sections

Key Result

lemma thmcounterlemma

The gradient $\nabla f$ of a convex and $L$-smooth function $f:\mathbb{X}\to \overline{\mathbb{R}}$ is $\frac{1}{L}$-co-coercive.

Figures (2)

  • Figure 1: Results for different values of $d$ and $n$.
  • Figure 2: Visualization of test loss dynamics and sample perturbation under the isHI updates.

Theorems & Definitions (27)

  • lemma thmcounterlemma: bauschke2011convex
  • lemma thmcounterlemma
  • theorem 1
  • corollary thmcountercorollary
  • corollary thmcountercorollary
  • remark thmcounterremark
  • theorem 2
  • lemma thmcounterlemma
  • corollary thmcountercorollary
  • corollary thmcountercorollary
  • ...and 17 more