Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Single-Loop Gradient Algorithm for Pessimistic Bilevel Optimization via Smooth Approximation

Qichao Cao, Shangzhi Zeng, Jin Zhang

TL;DR

This work addresses the challenge of Pessimistic Bilevel Optimization (PBO) by introducing a smooth, differentiable surrogate $\phi_{\rho,\sigma}(x)$ via penalization and regularization, enabling fully first-order gradient methods. Building on this, the authors propose SiPBA, a single-loop gradient algorithm that uses a one-step ascent-descent to approximate the saddle point and an inexact gradient to update the upper-level variable, avoiding second-order derivatives and inner-loop subproblem solves. Theoretical results establish epi-convergence of the smooth approximation to the original PBO and non-asymptotic convergence rates for SiPBA, along with a practical parameter schedule. Empirical evaluations on synthetic problems, spam classification, and hyper-representation tasks demonstrate competitive performance and efficiency, highlighting SiPBA’s robustness to hyperparameters and problem scale. Overall, the paper advances scalable, robust optimization under worst-case follower responses with potential impact on adversarial learning and robust hyperparameter tuning.

Abstract

Bilevel optimization has garnered significant attention in the machine learning community recently, particularly regarding the development of efficient numerical methods. While substantial progress has been made in developing efficient algorithms for optimistic bilevel optimization, the study of methods for solving Pessimistic Bilevel Optimization (PBO) remains relatively less explored, especially the design of fully first-order, single-loop gradient-based algorithms. This paper aims to bridge this research gap. We first propose a novel smooth approximation to the PBO problem, using penalization and regularization techniques. Building upon this approximation, we then propose SiPBA (Single-loop Pessimistic Bilevel Algorithm), a new gradient-based method specifically designed for PBO which avoids second-order derivative information or inner-loop iterations for subproblem solving. We provide theoretical validation for the proposed smooth approximation scheme and establish theoretical convergence for the algorithm SiPBA. Numerical experiments on synthetic examples and practical applications demonstrate the effectiveness and efficiency of SiPBA.

A Single-Loop Gradient Algorithm for Pessimistic Bilevel Optimization via Smooth Approximation

TL;DR

This work addresses the challenge of Pessimistic Bilevel Optimization (PBO) by introducing a smooth, differentiable surrogate via penalization and regularization, enabling fully first-order gradient methods. Building on this, the authors propose SiPBA, a single-loop gradient algorithm that uses a one-step ascent-descent to approximate the saddle point and an inexact gradient to update the upper-level variable, avoiding second-order derivatives and inner-loop subproblem solves. Theoretical results establish epi-convergence of the smooth approximation to the original PBO and non-asymptotic convergence rates for SiPBA, along with a practical parameter schedule. Empirical evaluations on synthetic problems, spam classification, and hyper-representation tasks demonstrate competitive performance and efficiency, highlighting SiPBA’s robustness to hyperparameters and problem scale. Overall, the paper advances scalable, robust optimization under worst-case follower responses with potential impact on adversarial learning and robust hyperparameter tuning.

Abstract

Bilevel optimization has garnered significant attention in the machine learning community recently, particularly regarding the development of efficient numerical methods. While substantial progress has been made in developing efficient algorithms for optimistic bilevel optimization, the study of methods for solving Pessimistic Bilevel Optimization (PBO) remains relatively less explored, especially the design of fully first-order, single-loop gradient-based algorithms. This paper aims to bridge this research gap. We first propose a novel smooth approximation to the PBO problem, using penalization and regularization techniques. Building upon this approximation, we then propose SiPBA (Single-loop Pessimistic Bilevel Algorithm), a new gradient-based method specifically designed for PBO which avoids second-order derivative information or inner-loop iterations for subproblem solving. We provide theoretical validation for the proposed smooth approximation scheme and establish theoretical convergence for the algorithm SiPBA. Numerical experiments on synthetic examples and practical applications demonstrate the effectiveness and efficiency of SiPBA.

Paper Structure

This paper contains 34 sections, 23 theorems, 65 equations, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contribution
  3. Contribution
  4. Related work
  5. Smooth approximation of PBO
  6. Smooth approximation of $\phi(x)$
  7. Asymptotic convergence of the approximation
  8. Single-loop gradient-based algorithm
  9. Convergence analysis
  10. Numerical experiments
  11. Synthetic example
  12. Spam classification
  13. Hyper-representation
  14. Linear hyper-representation on synthetic data
  15. Deep hyper-representation on MNIST and FashionMNIST
  16. ...and 19 more sections

Key Result

Theorem 2.1

Let $\rho, \sigma >0$ be given constants. Then, for any $x \in X$, $\phi_{\rho,\sigma}(x)$ is differentiable. Its gradient is given by: where $(y_{\rho,\sigma}^*(x),z_{\rho,\sigma}^*(x))$ is the unique saddle point for the minimax problem defining $\phi_{\rho,\sigma}(x)$ in phi.

Figures (3)

  • Figure 1: (a): Convergence curves of SiPBA, AdaProx-PD and AdaProx-SG on \ref{['constrainedtoy']} with $n=100$; (b): Iterations and runtime required for SiPBA on \ref{['constrainedtoy']} for varying problem dimensions $n$.
  • Figure 2: Test loss v.s. time in Hyper-representation with varying dimensions and noise levels.
  • Figure 3: Hyper-representation on MNIST and FashionMNIST.

Theorems & Definitions (25)

  • Theorem 2.1
  • Lemma 2.2
  • Proposition 2.3
  • Lemma 2.4
  • Theorem 2.5
  • Theorem 2.6
  • Proposition 4.1
  • Theorem 4.2
  • Corollary 4.3
  • Lemma B.1
  • ...and 15 more