Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Operator Splitting with Hamilton-Jacobi-based Proximals

Nicholas Di, Eric C. Chi, Samy Wu Fung

TL;DR

This work addresses the bottleneck of unavailable proximal operators in first-order optimization by introducing Hamilton-Jacobi-based proximals (HJ-Prox) and embedding them into operator splitting. It develops a unified convergence framework for zeroth-order variants of PPM, PGD, DRS, DYS, and PDHG, with almost-sure convergence under mild regularity and explicit Monte Carlo error bounds. Empirical results on LASSO, trend filtering, TV denoising, multitask learning, and genomics demonstrate that HJ-Prox matches analytic baselines and enables solving non-proximable problems directly on the original objective. The approach reduces implementation complexity by avoiding inner solvers or dual reformulations and supports hybrid strategies that preserve exact proximals when available while approximating only the non-proximable components.

Abstract

Operator splitting algorithms are a cornerstone of modern first-order optimization, decomposing complex problems into simpler subproblems solved via proximal operators. However, most functions lack closed-form proximal operators, which has long restricted these methods to a narrow set of problems. Hamilton-Jacobi-based proximal operator (HJ-Prox) is a recent derivative-free Monte Carlo technique based on Hamilton-Jacobi PDE theory, that approximates proximal operators numerically. In this work, we introduce a unified framework for operator splitting via HJ-Prox, which allows for deployment of operator splitting even when functions are not proximable. We prove that replacing exact proximal steps with HJ-Prox in algorithms such as proximal point, proximal gradient descent, Douglas-Rachford splitting, Davis-Yin splitting, and primal-dual hybrid gradient preserves convergence guarantees under mild assumptions. Numerical experiments demonstrate HJ-Prox is competitive and effective on a wide variety of statistical learning tasks.

Operator Splitting with Hamilton-Jacobi-based Proximals

TL;DR

This work addresses the bottleneck of unavailable proximal operators in first-order optimization by introducing Hamilton-Jacobi-based proximals (HJ-Prox) and embedding them into operator splitting. It develops a unified convergence framework for zeroth-order variants of PPM, PGD, DRS, DYS, and PDHG, with almost-sure convergence under mild regularity and explicit Monte Carlo error bounds. Empirical results on LASSO, trend filtering, TV denoising, multitask learning, and genomics demonstrate that HJ-Prox matches analytic baselines and enables solving non-proximable problems directly on the original objective. The approach reduces implementation complexity by avoiding inner solvers or dual reformulations and supports hybrid strategies that preserve exact proximals when available while approximating only the non-proximable components.

Abstract

Operator splitting algorithms are a cornerstone of modern first-order optimization, decomposing complex problems into simpler subproblems solved via proximal operators. However, most functions lack closed-form proximal operators, which has long restricted these methods to a narrow set of problems. Hamilton-Jacobi-based proximal operator (HJ-Prox) is a recent derivative-free Monte Carlo technique based on Hamilton-Jacobi PDE theory, that approximates proximal operators numerically. In this work, we introduce a unified framework for operator splitting via HJ-Prox, which allows for deployment of operator splitting even when functions are not proximable. We prove that replacing exact proximal steps with HJ-Prox in algorithms such as proximal point, proximal gradient descent, Douglas-Rachford splitting, Davis-Yin splitting, and primal-dual hybrid gradient preserves convergence guarantees under mild assumptions. Numerical experiments demonstrate HJ-Prox is competitive and effective on a wide variety of statistical learning tasks.
Paper Structure (31 sections, 17 theorems, 164 equations, 5 figures)

This paper contains 31 sections, 17 theorems, 164 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Contributions
  3. Background
  4. Hamilton-Jacobi-based Proximal (HJ-Prox)
  5. HJ-Prox-based Operator Splitting
  6. Existing Results
  7. Convergence Analysis of HJ-Prox-based Splitting
  8. Why Splitting?
  9. HJ Prox Discussion
  10. Experiments
  11. Comparison on Problems with Closed-Form Proximal Steps
  12. Comparison on Problems Without Closed-Form Proximals
  13. Splitting vs Non-splitting
  14. Overlapping Group LASSO Gene Expression
  15. Limitations and Future Work
  16. ...and 16 more sections

Key Result

Theorem 3.1

Let $\{x_k\}_{k\geq0}$ be a sequence in $\mathbb{R}^n$ generated by the iteration where each $T_k \colon \mathbb{R}^n \to \mathbb{R}^n$ belongs to the class of averaged operators, and the solution set $S = \bigcap_{k \geq 0} \mathrm{Fix}\, T_k$ is nonempty. If the error sequence is summable ($\sum_{k=0}^{\infty} \|\epsilon_k\| < \infty$) and the operators satisfy a closedness con

Figures (5)

  • Figure 1: Top row: LASSO experiment solved using PGD. HJ-Prox is called on the $\ell_1$ regularizer. Middle row: Sparse Group LASSO using DYS. HJ-Prox is called on the Group and $\ell_1$ regularizer. Bottom row: Total Variation Denoising solved with PDHG. HJ-Prox is called on the Total Variation penalty. All examples feature closed-form projection-based proximal operators. While final reconstructions are visually identical, the convergence rates differ, with the more difficult TV objective requiring increased iterations for the HJ-Prox variant.
  • Figure 2: Top row: Trend Filtering experiment solved using DRS. HJ-Prox is called on the differencing matrix. Bottom row: Multitask Learning using DRS. HJ-Prox is called on the clumped data fidelity term and onto the row and column penalties. Results confirm that HJ-Prox matches the convergence of complex baseline solvers, both of which require inner loops or problem reformulations.
  • Figure 3: Top row: HJ-PPM struggles to match convergence of HJ-PGD. Bottom row: HJ-PPM struggles to converge to similar solutions as opposed to using splitting methods. These experiments highlight that HJ-PPM is computationally less efficient and lacks the robust error control of the alternative methods.
  • Figure 4: Under a fixed $\delta$, pure PPM decreases the objective rapidly but is limited to a solution neighborhood with high error floors. The results further demonstrate that a single call of HJ-Prox outperforms two calls, suggesting a hybrid framework can be optimal when applicable. All fixed-point residuals are calculated using the analytical fixed-point operator to ensure fair comparison across methods.
  • Figure 5: Although similar in structure to the Group LASSO experiment in Figure \ref{['fig: LASSO, SGLASSO, TV']}, we allow the $\beta$ groupings to overlap here. We show convergence in log scale of the objective functions. We utilize an adaptive DYS here PedregosaGidel2018ICML. DYS-HJ is run with a decreasing $\delta$-schedule and fixed $N=1000$. FoGLasso is implemented exactly as in YuanLiuYe2011. We run for $100000$ iterations but zoom in to show the relevant iteration window for comparable convergence progress. The coefficient vector of length $13237$ has high correlation between FoGLASSO and DYS-HJ, and the group norms are numerically close together.

Theorems & Definitions (29)

  • Theorem 3.1: Convergence of Perturbed Krasnosel'skiĭ-Mann Iterates Combettes2001
  • Theorem 3.2: Error Bound on HJ-Prox crandall1983viscosity
  • Theorem 3.3: Monte Carlo Bound on HJ-Prox
  • Theorem 3.5: HJ-Prox PPM
  • Theorem 3.6: HJ-Prox PGD
  • Theorem 3.8: HJ-Prox DRS
  • Theorem 3.9: HJ-Prox DYS
  • Theorem 3.10: HJ-Prox PDHG
  • proof
  • proof
  • ...and 19 more