Randomized Feasibility Methods for Constrained Optimization with Adaptive Step Sizes

TL;DR

The paper tackles constrained convex optimization where the feasible set is an intersection of convex lower-level sets, handled by randomized feasibility updates that sample constraints and apply Polyak-type steps together with gradient methods on the objective. For strongly convex, Lipschitz-smooth objectives, it proves linear convergence in expectation with adaptive step sizes; for convex (possibly nonsmooth) objectives, it introduces a fully parameter-free scheme (DoWS) achieving an $O(1/\sqrt{T})$ worst-case rate in expectation. Infeasibility decays geometrically with randomized feasibility updates, and averaged iterates attain provable function-value bounds that depend on the distribution of the random constraint samples. Simulations on QCQP and SVM demonstrate the computational efficiency and robustness of the proposed methods relative to state-of-the-art approaches, particularly in large-scale or online settings where full constraint projections are expensive.

Abstract

We consider minimizing an objective function subject to constraints defined by the intersection of lower-level sets of convex functions. We study two cases: (i) strongly convex and Lipschitz-smooth objective function and (ii) convex but possibly nonsmooth objective function. To deal with the constraints that are not easy to project on, we use a randomized feasibility algorithm with Polyak steps and a random number of sampled constraints per iteration, while taking (sub)gradient steps to minimize the objective function. For case (i), we prove linear convergence in expectation of the objective function values to any prescribed tolerance using an adaptive stepsize. For case (ii), we develop a fully problem parameter-free and adaptive stepsize scheme that yields an $O(1/\sqrt{T})$ worst-case rate in expectation. The infeasibility of the iterates decreases geometrically with the number of feasibility updates almost surely, while for the averaged iterates, we establish an expected lower bound on the function values relative to the optimal value that depends on the distribution for the random number of sampled constraints. For certain choices of sample-size growth, optimal rates are achieved. Finally, simulations on a Quadratically Constrained Quadratic Programming (QCQP) problem and Support Vector Machines (SVM) demonstrate the computational efficiency of our algorithm compared to other state-of-the-art methods.

Figures (4)

• Figure 1: Simulation plots for the QCQP problem for the three cases considered. We show the decay in the function values as well as the infeasibility for all the algorithms. In the cases where the optimal function value $f^*$ is unknown, we use the value returned by the CVXPY solver as $f^*$. We choose $N_k = \lceil k^{1/2} \rceil$.
• Figure 2: Test misclassification error for Algorithms \ref{['algo_dows_feas']} and \ref{['algo_tdows_feas']}, and the primal-dual method with step sizes chosen via $3$-fold cross-validation.
• Figure 3: Simulation plots for the QCQP problem which is also presented in Section \ref{['sec:simulation']}.
• Figure 4: Objective function value, training feasibility violation, and test misclassification error for Algorithms \ref{['algo_dows_feas']} and \ref{['algo_tdows_feas']}, and the primal–dual method with step sizes chosen via $3$-fold cross-validation.

Theorems & Definitions (25)

• Lemma 3.1
• Lemma 4.3
• Theorem 4.4
• Theorem 5.3
• Theorem 5.5
• Lemma 2.1
• proof
• Lemma 2.2
• proof
• Lemma 2.3