Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Near-Optimal Convex Simple Bilevel Optimization with a Bisection Method

Jiulin Wang, Xu Shi, Rujun Jiang

TL;DR

This paper proposes a bisection algorithm to find a solution that is $\epsilon_f$-optimal for the upper-level objective and $\epsilon_g$-optimal for the lower-level objective and achieves a near-optimal rate.

Abstract

This paper studies a class of simple bilevel optimization problems where we minimize a composite convex function at the upper-level subject to a composite convex lower-level problem. Existing methods either provide asymptotic guarantees for the upper-level objective or attain slow sublinear convergence rates. We propose a bisection algorithm to find a solution that is $ε_f$-optimal for the upper-level objective and $ε_g$-optimal for the lower-level objective. In each iteration, the binary search narrows the interval by assessing inequality system feasibility. Under mild conditions, the total operation complexity of our method is ${\tilde {\mathcal{O}}}\left(\max\{\sqrt{L_{f_1}/ε_f},\sqrt{L_{g_1}/ε_g} \} \right)$. Here, a unit operation can be a function evaluation, gradient evaluation, or the invocation of the proximal mapping, $L_{f_1}$ and $L_{g_1}$ are the Lipschitz constants of the upper- and lower-level objectives' smooth components, and ${\tilde {\mathcal{O}}}$ hides logarithmic terms. Our approach achieves a near-optimal rate, matching the optimal rate in unconstrained smooth or composite convex optimization when disregarding logarithmic terms. Numerical experiments demonstrate the effectiveness of our method.

Near-Optimal Convex Simple Bilevel Optimization with a Bisection Method

TL;DR

This paper proposes a bisection algorithm to find a solution that is -optimal for the upper-level objective and -optimal for the lower-level objective and achieves a near-optimal rate.

Abstract

This paper studies a class of simple bilevel optimization problems where we minimize a composite convex function at the upper-level subject to a composite convex lower-level problem. Existing methods either provide asymptotic guarantees for the upper-level objective or attain slow sublinear convergence rates. We propose a bisection algorithm to find a solution that is -optimal for the upper-level objective and -optimal for the lower-level objective. In each iteration, the binary search narrows the interval by assessing inequality system feasibility. Under mild conditions, the total operation complexity of our method is . Here, a unit operation can be a function evaluation, gradient evaluation, or the invocation of the proximal mapping, and are the Lipschitz constants of the upper- and lower-level objectives' smooth components, and hides logarithmic terms. Our approach achieves a near-optimal rate, matching the optimal rate in unconstrained smooth or composite convex optimization when disregarding logarithmic terms. Numerical experiments demonstrate the effectiveness of our method.
Paper Structure (24 sections, 7 theorems, 53 equations, 4 figures, 1 table, 3 algorithms)

This paper contains 24 sections, 7 theorems, 53 equations, 4 figures, 1 table, 3 algorithms.

Table of Contents

  1. INTRODUCTION
  2. Our Approach
  3. Related Work
  4. PRELIMINARIES
  5. Assumptions
  6. MAIN ALGORITHM AND CONVERGENCE ANALYSIS
  7. Bisection Method
  8. Convergence Analysis under Assumption \ref{['as1']}
  9. CONVERGENCE ANALYSIS UNDER OTHER ASSUMPTIONS
  10. Convergence Analysis under Hölderian Error Bound Assumption
  11. Convergence Analysis under an Additional Assumption on the Optimal Value of (\ref{['pb:BP2']})
  12. NUMERICAL EXPERIMENTS
  13. Minimum Norm Solution Problem (MNP)
  14. Logistic Regression Problem (LRP)
  15. CONCLUSION
  16. ...and 9 more sections

Key Result

Lemma 1

For any fixed $c$, if Condition (cond1) is satisfied, then System (system1) is infeasible, and $c$ is a lower bound of $p^*$. If Condition (cond1) is not satisfied, then we can obtain $\tilde{{\mathbf x}}_c$ as an $\epsilon_g$-optimal solution of the lower-level problem and $f(\tilde{{\mathbf x}}_c)

Figures (4)

  • Figure 1: Variation of $\bar{g}(c)$ over $(f^{*},+\infty)$
  • Figure 2: The performance of Bisec-BiO compared with other methods in MNP.
  • Figure 3: The performance of Bisec-BiO compared with other methods in LRP.
  • Figure 4: The performance of Bisec-BiO compared with other methods in SSP.

Theorems & Definitions (21)

  • Definition 1
  • Remark 1
  • Remark 2
  • Lemma 1
  • Theorem 1
  • Remark 3
  • Proposition 1
  • Corollary 1
  • Example 1: A toy example
  • Proposition 2
  • ...and 11 more