Differentially Private Bilevel Optimization: Efficient Algorithms with Near-Optimal Rates
Andrew Lowy, Daogao Liu
TL;DR
The paper addresses privacy-preserving bilevel optimization (BLO) by developing DP algorithms and minimax-rate bounds for both convex and nonconvex outer objectives, focusing on BLO-ERM and BLO-SO. It leverages the exponential mechanism and regularized exponential mechanism for DP, and introduces robust log-concave sampling under inexact function evaluations to implement these mechanisms efficiently in BLO. For the convex setting, it provides nearly tight upper and lower bounds that nearly match single-level DP-ERM up to bilevel-complexity terms, and for the nonconvex setting, it achieves state-of-the-art, dimension-independent rates via a second-order method and warm-start strategy. The work offers a comprehensive theoretical foundation for DP BLO with practical algorithmic designs, paving the way for privacy-preserving hierarchical learning tasks, though it calls for empirical validation and further improvements in scalability and SO settings.
Abstract
Bilevel optimization, in which one optimization problem is nested inside another, underlies many machine learning applications with a hierarchical structure -- such as meta-learning and hyperparameter optimization. Such applications often involve sensitive training data, raising pressing concerns about individual privacy. Motivated by this, we study differentially private bilevel optimization. We first focus on settings where the outer-level objective is convex, and provide novel upper and lower bounds on the excess empirical risk for both pure and approximate differential privacy. These bounds are nearly tight and essentially match the optimal rates for standard single-level differentially private ERM, up to additional terms that capture the intrinsic complexity of the nested bilevel structure. We also provide population loss bounds for bilevel stochastic optimization. The bounds are achieved in polynomial time via efficient implementations of the exponential and regularized exponential mechanisms. A key technical contribution is a new method and analysis of log-concave sampling under inexact function evaluations, which may be of independent interest. In the non-convex setting, we develop novel algorithms with state-of-the-art rates for privately finding approximate stationary points. Notably, our bounds do not depend on the dimension of the inner problem.