Moreau Envelope for Nonconvex Bi-Level Optimization: A Single-loop and Hessian-free Solution Strategy

TL;DR

This paper tackles large-scale bi-level optimization with nonconvex lower-level objectives by introducing a Moreau envelope reformulation that enables a single-loop, Hessian-free gradient method (MEHA) using only first-order information. It provides non-asymptotic convergence guarantees and explicit rates, leveraging weak convexity to avoid Hessian computations while maintaining theoretical rigor. The core contributions include the MEHA algorithm, a proximal-gradient-based update scheme, and a thorough non-asymptotic analysis that yields rates such as $\min_{0\le k\le K} \|\theta^{k}-\theta_{\gamma}^*(x^{k},y^{k})\| = O(K^{-1/2})$ and $\min_{0\le k\le K} R_k(x^{k+1},y^{k+1}) = O(K^{-(1-2p)/2})$; with a fixed penalty, a $O(K^{-1/2})$ rate persists in the objective. Empirically, MEHA demonstrates superior speed and scalability on synthetic BLO tasks, two hyperparameter learning problems, and neural architecture search, validating its practical impact for large-scale, nonconvex BLO settings. Overall, the work advances BLO by delivering a practical, Hessian-free, theoretically grounded, single-loop method suitable for modern ML workloads.

Abstract

This work focuses on addressing two major challenges in the context of large-scale nonconvex Bi-Level Optimization (BLO) problems, which are increasingly applied in machine learning due to their ability to model nested structures. These challenges involve ensuring computational efficiency and providing theoretical guarantees. While recent advances in scalable BLO algorithms have primarily relied on lower-level convexity simplification, our work specifically tackles large-scale BLO problems involving nonconvexity in both the upper and lower levels. We simultaneously address computational and theoretical challenges by introducing an innovative single-loop gradient-based algorithm, utilizing the Moreau envelope-based reformulation, and providing non-asymptotic convergence analysis for general nonconvex BLO problems. Notably, our algorithm relies solely on first-order gradient information, enhancing its practicality and efficiency, especially for large-scale BLO learning tasks. We validate our approach's effectiveness through experiments on various synthetic problems, two typical hyper-parameter learning tasks, and a real-world neural architecture search application, collectively demonstrating its superior performance.

Figures (6)

• Figure 1: Illustrating the convergence curves of advanced BLO schemes and MEHA by the criteria, $\|x-x^{*}\|/\|x^{*}\|$ and $\|y-y^{*}\|/\|y^{*}\|$, under LL merely convex case.
• Figure 2: Convergence curves of advanced BLO methods and MEHA by the criteria, $\|x-x^{*}\|/\|x^{*}\|$, under the LL non-convex case with different dimensions (10, 50, 100, and 200).
• Figure 3: Comparison of data hyper-cleaning on FashionMNIST.
• Figure 4: Illustrating the convergence curves of advanced BLO methods and MEHA by the criterion of $\|x-x^{*}\|/\|x^{*}\|$ and $\|y-y^{*}\|/\|y^{*}\|$ under LL strong convex case.
• Figure 5: Visualizing the convergence behaviours of BOME, BVFIM, IAPTT and MEHA under LL non-convex case with one dimension, using the metrics of descent direction $\|\mathbf{d}x\|$, UL objective $F$ and reconstruction error with $x$.

Theorems & Definitions (34)

• Lemma 3.3
• Theorem 3.4
• Remark 3.5
• Theorem 3.6
• Lemma 1.1
• Theorem 1.2
• proof
• Lemma 1.3
• Theorem 1.4
• proof