Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Proximal Oracles for Optimization and Sampling

Jiaming Liang, Yongxin Chen

TL;DR

This work develops a unified proximal framework for both optimization and sampling when the objective/potential is convex but non-smooth, focusing on Hölder-smooth and hybrid Hölder components. It introduces a regularized cutting-plane-based proximal map (PBS) and a universal adaptive proximal bundle method (APBM) to solve proximal subproblems without problem-parameter input. For sampling, it constructs a proximal sampling oracle via rejection sampling anchored by PBS and integrates it with the alternating sampling framework (ASF) to obtain non-asymptotic convergence guarantees for log-concave distributions with non-smooth potentials. The results connect proximal optimization and sampling through RGO/ASF, delivering dimension-sensitive complexity bounds and providing practical, universal algorithms for both Hölder-smooth and hybrid settings.

Abstract

We consider convex optimization with non-smooth objective function and log-concave sampling with non-smooth potential (negative log density). In particular, we study two specific settings where the convex objective/potential function is either Hölder smooth or in hybrid form as the finite sum of Hölder smooth components. To overcome the challenges caused by non-smoothness, our algorithms employ two powerful proximal frameworks in optimization and sampling: the proximal point framework for optimization and the alternating sampling framework (ASF) that uses Gibbs sampling on an augmented distribution. A key component of both optimization and sampling algorithms is the efficient implementation of the proximal map by the regularized cutting-plane method. We establish its iteration-complexity under both Hölder smoothness and hybrid settings using novel convergence analysis, yielding results that are new to the literature. We further propose an adaptive proximal bundle method for non-smooth optimization that employs an aggressive adaptive stepsize strategy, which adjusts stepsizes only when necessary and never rejects iterates. The proposed method is universal since it does not need any problem parameters as input. Additionally, we provide an exact implementation of a proximal sampling oracle, analogous to the proximal map in optimization, along with simple complexity analyses for both the Hölder smooth and hybrid cases, using a novel technique based on a modified Gaussian integral. Finally, we combine this proximal sampling oracle and ASF to obtain a Markov chain Monte Carlo method with non-asymptotic complexity bounds for sampling in Hölder smooth and hybrid settings.

Proximal Oracles for Optimization and Sampling

TL;DR

This work develops a unified proximal framework for both optimization and sampling when the objective/potential is convex but non-smooth, focusing on Hölder-smooth and hybrid Hölder components. It introduces a regularized cutting-plane-based proximal map (PBS) and a universal adaptive proximal bundle method (APBM) to solve proximal subproblems without problem-parameter input. For sampling, it constructs a proximal sampling oracle via rejection sampling anchored by PBS and integrates it with the alternating sampling framework (ASF) to obtain non-asymptotic convergence guarantees for log-concave distributions with non-smooth potentials. The results connect proximal optimization and sampling through RGO/ASF, delivering dimension-sensitive complexity bounds and providing practical, universal algorithms for both Hölder-smooth and hybrid settings.

Abstract

We consider convex optimization with non-smooth objective function and log-concave sampling with non-smooth potential (negative log density). In particular, we study two specific settings where the convex objective/potential function is either Hölder smooth or in hybrid form as the finite sum of Hölder smooth components. To overcome the challenges caused by non-smoothness, our algorithms employ two powerful proximal frameworks in optimization and sampling: the proximal point framework for optimization and the alternating sampling framework (ASF) that uses Gibbs sampling on an augmented distribution. A key component of both optimization and sampling algorithms is the efficient implementation of the proximal map by the regularized cutting-plane method. We establish its iteration-complexity under both Hölder smoothness and hybrid settings using novel convergence analysis, yielding results that are new to the literature. We further propose an adaptive proximal bundle method for non-smooth optimization that employs an aggressive adaptive stepsize strategy, which adjusts stepsizes only when necessary and never rejects iterates. The proposed method is universal since it does not need any problem parameters as input. Additionally, we provide an exact implementation of a proximal sampling oracle, analogous to the proximal map in optimization, along with simple complexity analyses for both the Hölder smooth and hybrid cases, using a novel technique based on a modified Gaussian integral. Finally, we combine this proximal sampling oracle and ASF to obtain a Markov chain Monte Carlo method with non-asymptotic complexity bounds for sampling in Hölder smooth and hybrid settings.
Paper Structure (14 sections, 24 theorems, 124 equations, 2 figures, 1 table, 4 algorithms)

This paper contains 14 sections, 24 theorems, 124 equations, 2 figures, 1 table, 4 algorithms.

Table of Contents

  1. Introduction
  2. Proximal Optimization and Sampling
  3. Example.
  4. Algorithm and Complexities for the Proximal Subproblem
  5. Complexity for Hölder Smooth Optimization
  6. Complexity for Hybrid Optimization
  7. Implementation of Algorithm \ref{['alg:PBS']}
  8. Adaptive Proximal Bundle Method
  9. Proximal Sampling Algorithm
  10. Sampling from Hölder Smooth Potentials
  11. Sampling from Hybrid Potentials
  12. Conclusions
  13. Technical results
  14. Missing proofs in Subsection \ref{['sec:composite']}

Key Result

Lemma 3.1

Assume $f$ is convex. For every $j\ge 1$, define Let $J, x_J, \tilde{x}_J$ be the outputs of Algorithm alg:PBS, then the following statements hold:

Figures (2)

  • Figure 1: Proximal subproblem progress of Algorithm \ref{['alg:PBS']} in quadratic programming.
  • Figure 2: Proximal subproblem progress of Algorithm \ref{['alg:PBS']} in $\ell_p$ regression.

Theorems & Definitions (43)

  • Definition 2.1
  • Lemma 3.1
  • proof
  • Lemma 3.2
  • proof
  • Proposition 3.3
  • proof
  • Lemma 3.4
  • proof
  • Theorem 3.5
  • ...and 33 more