Efficient algorithms for implementing incremental proximal-point methods

TL;DR

The paper tackles the practical challenge of implementing incremental proximal-point methods by introducing a duality-based framework for convex onto linear compositions. By decomposing goals into a convex outer function $h$ and a simple inner structure, it derives modular procedures to compute proximal operators through convex conjugates and Moreau envelopes, enabling efficient, reusable software components. The authors provide concrete implementations for common losses (linear regression, logistic regression, APROX, and prox-linear) and extend the framework to regularized and mini-batch settings, supported by empirical speed and stability assessments. The accompanying open-source Python library demonstrates that proximal-point updates can be competitive with stochastic gradient steps in practice while offering robustness to step-size choices, thereby lowering hyper-parameter tuning costs and encouraging broader experimentation.

Abstract

Model training algorithms which observe a small portion of the training set in each computational step are ubiquitous in practical machine learning, and include both stochastic and online optimization methods. In the vast majority of cases, such algorithms typically observe the training samples via the gradients of the cost functions the samples incur. Thus, these methods exploit are the slope of the cost functions via their first-order approximations. To address limitations of gradient-based methods, such as sensitivity to step-size choice in the stochastic setting, or inability to use small function variability in the online setting, several streams of research attempt to exploit more information about the cost functions than just their gradients via the well-known proximal operators. However, implementing such methods in practice poses a challenge, since each iteration step boils down to computing the proximal operator, which may not be easy. In this work we devise a novel algorithmic framework, which exploits convex duality theory to achieve both algorithmic efficiency and software modularity of proximal operator implementations, in order to make experimentation with incremental proximal optimization algorithms accessible to a larger audience of researchers and practitioners, by reducing the gap between their theoretical description in research papers and their use in practice. We provide a reference Python implementation for the framework developed in this paper as an open source library at on https://github.com/alexshtf/inc_prox_pt/releases/tag/prox_pt_paper, along with examples which demonstrate our implementation on a variety of problems, and reproduce the numerical experiments in this paper. The pure Python reference implementation is not necessarily the most efficient, but is a basis for creating efficient implementations by combining Python with a native backend.

Figures (6)

• Figure 1: Execution speed evaluation of incremental proximal point. Each point is a timing of a pair of experiments on the same problem, where the $x$ coordinate is the execution time of one SGD epoch, whereas the $y$ coordinate is the execution time of one proximal point epoch. The corresponding line is a linear regression line, with its slope $\alpha$ labeled, to appreciate the ratio between the SGD and proximal point execution times, on average. The columns are various problem dimensions, from 1000 to 6000, and the rows are various mini-batch sizes for the incremental gradient method. We can see by the first row, for example, that without mini-batching the proximal-point method is actually faster than an incremental gradient method ($\alpha > 0$) based on PyTorch's automatic differentiation. In the last row, for example, we can see that for batch sizes of 32 samples the incremental proximal point method is roughly 4 times slower for least-squares problems and 6 times slower for logistic regression problems compared to an incremental gradient method.
• Figure 2: Reproduction of the stability results of asi_duchi_better_models. Indeed, for both logistic regression and least squares problems, the training loss of an incremental proximal-point method is significantly more stable w.r.t the step-size choice, reinforcing the claim that with an incremental proximal-point method we can just "make an few educated guesses" instead of performing an extensive hyper-parameter search for the best step-size.
• Figure 3: Execution speed evaluation of incremental proximal point. Each point is a timing of a pair of experiments on the same problem, where the $x$ coordinate is the execution time of one SGD epoch, whereas the $y$ coordinate is the execution time of one proximal point epoch. The columns are various problem dimensions, and the rows are various mini-batch sizes for the incremental proximal gradient method. The columns are various problem dimensions, and the rows are various mini-batch sizes for the incremental proximal gradient method. We can see by the first row, for example, that without mini-batching the proximal-point method is 5-10 times slower than its proximal-gradient variant. In the last row, for example, we can see that for batch sizes of 32 samples the incremental proximal point method is roughly 20-150 times slower, depending on the problem and the dimension.
• Figure 4: Reproduction of the stability results of asi_duchi_better_models. Indeed, for both logistic regression and least squares problems, with both L1 and L2 regularization, the training loss of an incremental proximal-point method is significantly more stable w.r.t the step-size choice than the incremental proximal-gradient method, reinforcing the claim that with an incremental proximal-point method we can just "make an few educated guesses" instead of performing an extensive hyper-parameter search for the best step-size.
• Figure 5: Execution speed evaluation of mini-batch incremental proximal point. Each point is a timing of a pair of experiments on the same problem, where the $x$ coordinate is the execution time of one SGD epoch, whereas the $y$ coordinate is the execution time of one mini-batch proximal point epoch. Points differ in running times due to generated data-set size. The columns are various problem dimensions, and the rows are various mini-batch sizes for the incremental gradient method. The corresponding line is a least-squares regression line, whose slope allows to appreciate the ratio between the SGD and proximal point running times. For least-squares problems, the mini-batched proximal point method is on par with the incremental gradient method. However, logistic regression problems employ the Logistic class which incurs the overhead of the CVXPY framework and of a generic conic solver, and is approximately 15 times slower for mini-batches of 32 samples, and approximately 100 times slower for a mini-batch of 8 samples, where CVXPY's overhead is more significant.

Theorems & Definitions (9)

• theorem 1: Strong Duality
• proof
• proposition thmcounterproposition
• proof
• definition thmcounterdefinition: Moreau Envelope moreau_envelope
• proposition thmcounterproposition
• proof
• proposition thmcounterproposition
• proof