An Adaptive Parameter-free and Projection-free Restarting Level Set Method for Constrained Convex Optimization Under the Error Bound Condition
Qihang Lin, Negar Soheili, Runchao Ma, Selvaprabu Nadarajah
TL;DR
This work introduces a Restarting Level Set (RLS) method for constrained convex optimization that is both parameter-free and projection-free. By solving a family of level-set subproblems in parallel and restarting copies based on objective progress, RLS achieves adaptive acceleration under the error bound condition without requiring prior parameter knowledge or complex projections. Theoretical guarantees show that RLS attains $\epsilon$-optimal and $\epsilon$-feasible solutions with favorable iteration complexity, up to logarithmic factors, in both non-smooth and smooth settings. Empirical results on fairness-constrained and Neyman–Pearson classification tasks demonstrate competitive and often superior performance compared to state-of-the-art parameter-free baselines, highlighting the method’s practical impact for constrained optimization in machine learning and related fields.
Abstract
Recent efforts to accelerate first-order methods have focused on convex optimization problems that satisfy a geometric property known as error-bound condition, which covers a broad class of problems, including piece-wise linear programs and strongly convex programs. Parameter-free first-order methods that employ projection-free updates have the potential to broaden the benefit of acceleration. Such a method has been developed for unconstrained convex optimization but is lacking for general constrained convex optimization. We propose a parameter-free level-set method for the latter constrained case based on projection-free subgradient method that exhibits accelerated convergence for problems that satisfy an error-bound condition. Our method maintains a separate copy of the level-set sub-problem for each level parameter value and restarts the computation of these copies based on objective function progress. Applying such a restarting scheme in a level-set context is novel and results in an algorithm that dynamically adapts the precision of each copy. This property is key to extending prior restarting methods based on static precision that have been proposed for unconstrained convex optimization to handle constraints. We report promising numerical performance relative to benchmark methods.