Safe Gradient Flow for Bilevel Optimization
Sina Sharifi, Nazanin Abolfazli, Erfan Yazdandoost Hamedani, Mahyar Fazlyab
TL;DR
This work addresses the computational challenge of bilevel optimization by marrying a gradient-flow strategy for the upper-level objective with a safety filter that enforces the lower-level optimality constraints in a single loop. The core approach, termed safe gradient flow (SGF), projects the velocity onto a constraint-consistent manifold by solving a convex QP, yielding closed-form updates that guarantee forward invariance of the lower-level constraint set and convergence to a neighborhood of the bilevel optimum via Lyapunov analysis. To scale to high-dimensional lower-level problems, the authors derive an inversion-free variant that replaces Hessian inversions with a single-level reformulation using a barrier-inspired constraint $h(x,y)=\|\nabla_y g(x,y)\|^2$, plus a relaxed version with $h(x,y) \le \varepsilon^2$ and a controlled-proximity term, both accompanied by convergence guarantees. Theoretical results establish Lipschitz continuity of the projection-based dynamics and nonincreasing Lyapunov energy, while experiments on synthetic benchmarks and MNIST-based data hyper-cleaning demonstrate practical performance gains and robustness of the proposed methods. Overall, the paper provides a principled, controllable, and scalable framework for solving bilevel problems in a single loop with provable guarantees and empirical validation.
Abstract
Bilevel optimization is a key framework in hierarchical decision-making, where one problem is embedded within the constraints of another. In this work, we propose a control-theoretic approach to solving bilevel optimization problems. Our method consists of two components: a gradient flow mechanism to minimize the upper-level objective and a safety filter to enforce the constraints imposed by the lower-level problem. Together, these components form a safe gradient flow that solves the bilevel problem in a single loop. To improve scalability with respect to the lower-level problem's dimensions, we introduce a relaxed formulation and design a compact variant of the safe gradient flow. This variant minimizes the upper-level objective while ensuring the lower-level decision variable remains within a user-defined suboptimality. Using Lyapunov analysis, we establish convergence guarantees for the dynamics, proving that they converge to a neighborhood of the optimal solution. Numerical experiments further validate the effectiveness of the proposed approaches. Our contributions provide both theoretical insights and practical tools for efficiently solving bilevel optimization problems.