On iteratively regularized first-order methods for simple bilevel optimization
Sepideh Samadi, Daniel Burbano, Farzad Yousefian
TL;DR
This work addresses simple bilevel optimization (SBO) where the goal is to select, among the minimizers of a lower-level composite problem, a solution optimal for a secondary objective, formalized as $\min_x f(x)+\omega_f(x)$ subject to $x\in X^*_{\bar h}=\arg\min_x h(x)+\omega_h(x)$. It develops three iterative-regularization frameworks: (i) IR-ISTA$_s$ for a composite strongly convex upper level with asymptotic convergence and simultaneous sublinear rates (and linear rates under weak sharp minima); (ii) R-VFISTA$_s$, a regularized accelerated proximal method achieving quadratically decaying sublinear rates and improved linear convergence under sharpness; and (iii) IPR-VFISTA$_{nc}$ for a smooth nonconvex upper level, delivering convergence rate statements to stationary points. The paper also provides a nonconvex SBO treatment via an inexact projection onto the lower-level optimal set, with provable bounds and complexity improvements, and reports preliminary numerical experiments on ill-posed linear inverse problems. Overall, the results advance theory and practice for SBO by delivering simultaneous, nonasymptotic guarantees and accelerated convergence under broad structural assumptions. The findings have practical impact for robust solution selection in ill-posed settings and for designing efficient, provably convergent bilevel optimization algorithms.
Abstract
We consider simple bilevel optimization problems where the goal is to compute among the optimal solutions of a composite convex optimization problem, one that minimizes a secondary objective function. Our main contribution is threefold. (i) When the upper-level objective is a composite strongly convex function, we propose an iteratively regularized proximal gradient method in that the regularization parameter is updated at each iteration under a prescribed rule. We establish the asymptotic convergence of the generated iterate to the unique optimal solution. Further, we derive simultaneous sublinear convergence rates for suitably defined infeasibility and suboptimality error metrics. When the optimal solution set of the lower-level problem admits a weak sharp minimality condition, utilizing a constant regularization parameter, we show that this method achieves simultaneous linear convergence rates. (ii) For addressing the setting in (i), we also propose a regularized accelerated proximal gradient method. We derive quadratically decaying sublinear convergence rates for both infeasibility and suboptimality error metrics. When weak sharp minimality holds, a linear convergence rate with an improved dependence on the condition number is achieved. (iii) When the upper-level objective is a smooth nonconvex function, we propose an inexactly projected iteratively regularized gradient method. Under suitable assumptions, we derive new convergence rate statements for computing a stationary point of the simple bilevel problem. We present preliminary numerical experiments for resolving three instances of ill-posed linear inverse problems.