Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Perturbed Value-Function-Based Interior-Point Method for Perturbed Pessimistic Bilevel Problems

Haimei Huo, Risheng Liu, Zhixun Su

TL;DR

A log-barrier function is introduced to replace the inequality constraint associated with the value function of the lower level problem, and approximating this value function, and an algorithm named Perturbed Value-Function-based Interior-point Method(PVFIM) is proposed.

Abstract

Bilevel optimizaiton serves as a powerful tool for many machine learning applications. Perturbed pessimistic bilevel problem PBP$ε$, with $ε$ being an arbitrary positive number, is a variant of the bilevel problem to deal with the case where there are multiple solutions in the lower level problem. However, the provably convergent algorithms for PBP$ε$ with a nonlinear lower level problem are lacking. To fill the gap, we consider in the paper the problem PBP$ε$ with a nonlinear lower level problem. By introducing a log-barrier function to replace the inequality constraint associated with the value function of the lower level problem, and approximating this value function, an algorithm named Perturbed Value-Function-based Interior-point Method(PVFIM) is proposed. We present a stationary condition for PBP$ε$, which has not been given before, and we show that PVFIM can converge to a stationary point of PBP$ε$. Finally, experiments are presented to verify the theoretical results and to show the application of the algorithm to GAN.

A Perturbed Value-Function-Based Interior-Point Method for Perturbed Pessimistic Bilevel Problems

TL;DR

A log-barrier function is introduced to replace the inequality constraint associated with the value function of the lower level problem, and approximating this value function, and an algorithm named Perturbed Value-Function-based Interior-point Method(PVFIM) is proposed.

Abstract

Bilevel optimizaiton serves as a powerful tool for many machine learning applications. Perturbed pessimistic bilevel problem PBP, with being an arbitrary positive number, is a variant of the bilevel problem to deal with the case where there are multiple solutions in the lower level problem. However, the provably convergent algorithms for PBP with a nonlinear lower level problem are lacking. To fill the gap, we consider in the paper the problem PBP with a nonlinear lower level problem. By introducing a log-barrier function to replace the inequality constraint associated with the value function of the lower level problem, and approximating this value function, an algorithm named Perturbed Value-Function-based Interior-point Method(PVFIM) is proposed. We present a stationary condition for PBP, which has not been given before, and we show that PVFIM can converge to a stationary point of PBP. Finally, experiments are presented to verify the theoretical results and to show the application of the algorithm to GAN.
Paper Structure (27 sections, 20 theorems, 156 equations, 4 figures, 2 tables, 2 algorithms)

This paper contains 27 sections, 20 theorems, 156 equations, 4 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Problem Setting
  4. Basic Tools
  5. Algorithm
  6. Approximate Minimax Problem
  7. A New Algorithm Named PVFIM for Problem PBP$\epsilon$ in (\ref{['eq4']})
  8. Theoretical Results
  9. Stationary Condition for Problem PBP$\epsilon$ in (\ref{['eq4']})
  10. Convergence of PVFIM in Algorithm \ref{['alg:1']}
  11. Experimental Results
  12. Synthetic Perturbed Pessimistic Bilevel Problems
  13. Generative Adversarial Networks
  14. Synthetic Data
  15. Real-World Data
  16. ...and 12 more sections

Key Result

Proposition 1

Suppose Assumptions assum:1 and assum:2 hold. Then $f^*(\boldsymbol{x})$ in (eq5) is differentiable on $\mathcal{X}$. Furthermore, for each $\boldsymbol{x}\in \mathcal{X}$, $\{\nabla_{\boldsymbol{x}}f(\boldsymbol{x}, \boldsymbol{y}): \boldsymbol{y}\in \mathcal{S}(\boldsymbol{x})\}$ is a single point where $\boldsymbol{y}_1^* \in \mathcal{S}(\boldsymbol{x})$, and $\mathcal{S}(\boldsymbol{x}) := { {

Figures (4)

  • Figure 1: Illustrating the convergence of PVFIM to the global optimal point of problem PBP$\epsilon$ in (\ref{['eq4']}) with different choices of $(T_l, J_l, K_l)$ for each positive integer $l$.
  • Figure 2: Illustrating the numerical performance of PVFIM with different choices of initial points.
  • Figure 3: Comparison of the generated samples which have the smallest Wasserstein distance to the target samples during the iterations. (a) samples generated by GAN.(b) samples generated by Unrolled GAN. (c) samples generated by PVFIM. (d) target samples.
  • Figure 4: Comparison of the IS(the higher, the better) and FID(the smaller, the better) over the iterations. (a), (b) present the results on the MNIST dataset, and (c), (d) present the results on the CIFAR10 dataset.

Theorems & Definitions (39)

  • Proposition 1
  • Definition 1
  • Definition 2
  • Example 1
  • Proposition 2
  • Proposition 3
  • Proposition 4
  • Theorem 1
  • Example 2
  • Definition 3
  • ...and 29 more