Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Hybrid Subgradient Method for Nonsmooth Nonconvex Bilevel Optimization

Nachuan Xiao, Xiaoyin Hu, Xin Liu, Kim-Chuan Toh

TL;DR

The paper tackles nonsmooth, nonconvex bilevel optimization by recasting the problem as a constrained optimization over the LL stationary manifold and exploiting mild regularity through LICQ-like conditions. It develops a two-timescale momentum-accelerated subgradient method (TMG) for local convergence, and an FRG feasibility restoration scheme to achieve global convergence, culminating in a novel hybrid algorithm that adaptively switches between phases. The analysis leverages a conservative-field framework and an exact penalty function $h_{\beta}$ to connect BLO to a single-level problem and to establish convergence of the interpolated processes to first-order stationary points. The results are supported by preliminary numerical experiments showing improved convergence speed and solution quality over existing methods, highlighting the practical potential for large-scale nonsmooth BLOs. The approach offers a scalable, Hessian-free framework with solid theoretical guarantees and broad applicability to problems in learning, optimization, and hyperparameter tuning.

Abstract

In this paper, we focus on the nonconvex-nonconvex bilevel optimization problem (BLO), where both upper-level and lower-level objectives are nonconvex, with the upper-level problem potentially being nonsmooth. We develop a two-timescale momentum-accelerated subgradient method (TMG) that employs two-timescale stepsizes, and establish its local convergence when initialized within a sufficiently small neighborhood of the feasible region. To develop a globally convergent algorithm for (BLO), we introduce a feasibility restoration scheme (FRG) that drives iterates toward the feasible region. Both (TMG) and (FRG) only require the first-order derivatives of the upper-level and lower-level objective functions, ensuring efficient computations in practice. We then develop a novel hybrid method that alternates between (TMG) and (FRG) and adaptively estimates its hyperparameters. Under mild conditions, we establish the global convergence properties of our proposed algorithm. Preliminary numerical experiments demonstrate the high efficiency and promising potential of our proposed algorithm.

A Hybrid Subgradient Method for Nonsmooth Nonconvex Bilevel Optimization

TL;DR

The paper tackles nonsmooth, nonconvex bilevel optimization by recasting the problem as a constrained optimization over the LL stationary manifold and exploiting mild regularity through LICQ-like conditions. It develops a two-timescale momentum-accelerated subgradient method (TMG) for local convergence, and an FRG feasibility restoration scheme to achieve global convergence, culminating in a novel hybrid algorithm that adaptively switches between phases. The analysis leverages a conservative-field framework and an exact penalty function to connect BLO to a single-level problem and to establish convergence of the interpolated processes to first-order stationary points. The results are supported by preliminary numerical experiments showing improved convergence speed and solution quality over existing methods, highlighting the practical potential for large-scale nonsmooth BLOs. The approach offers a scalable, Hessian-free framework with solid theoretical guarantees and broad applicability to problems in learning, optimization, and hyperparameter tuning.

Abstract

In this paper, we focus on the nonconvex-nonconvex bilevel optimization problem (BLO), where both upper-level and lower-level objectives are nonconvex, with the upper-level problem potentially being nonsmooth. We develop a two-timescale momentum-accelerated subgradient method (TMG) that employs two-timescale stepsizes, and establish its local convergence when initialized within a sufficiently small neighborhood of the feasible region. To develop a globally convergent algorithm for (BLO), we introduce a feasibility restoration scheme (FRG) that drives iterates toward the feasible region. Both (TMG) and (FRG) only require the first-order derivatives of the upper-level and lower-level objective functions, ensuring efficient computations in practice. We then develop a novel hybrid method that alternates between (TMG) and (FRG) and adaptively estimates its hyperparameters. Under mild conditions, we establish the global convergence properties of our proposed algorithm. Preliminary numerical experiments demonstrate the high efficiency and promising potential of our proposed algorithm.

Paper Structure

This paper contains 20 sections, 20 theorems, 96 equations, 2 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivations
  3. Regularity condition
  4. Two-timescale scheme
  5. Contributions
  6. Organization
  7. Preliminary
  8. Basic notations
  9. Conservative field and path-differentiable function
  10. Differential inclusion
  11. Global Convergence of Hybrid Method
  12. Basic assumptions, the hybrid method, and the main result
  13. Proof of Theorem \ref{['Theo_convergence_NS_joint_nondeg']}
  14. Basic properties of exact penalty function
  15. Convergence properties of subgradient method
  16. ...and 5 more sections

Key Result

Lemma 2.4

Given any convex-valued graph-closed set-valued mapping $\mathcal{D}$, any diminishing positive sequence $\{\delta_k\}$, any uniformly bounded points $\{{x_{k} }\}$, and any sequence $\{d_k\}$ that satisfies $d_k \in \mathrm{conv}\,\left( \mathcal{D}^{\delta_k}({x_{k} }) \right)$, we have that for a

Figures (2)

  • Figure 1: Test results on the performance of all the compared methods for synthetic example \ref{['Eq_NumExp_synthetic_1']}.
  • Figure 2: Test results on the performance of all the compared methods for the hyperparameter selection problem \ref{['Eq_NumExp_HS']}.

Theorems & Definitions (45)

  • Definition 2.1
  • Definition 2.2
  • Definition 2.3
  • Lemma 2.4
  • Definition 2.5
  • Definition 2.6: Definition II in benaim2005stochastic
  • Definition 2.7
  • Lemma 2.8
  • Remark 3.2
  • Remark 3.4
  • ...and 35 more