Regularized Federated Methods with Universal Guarantees for Simple Bilevel Optimization
Mohammadjavad Ebrahimi, Yuyang Qiu, Shisheng Cui, Farzad Yousefian
TL;DR
The paper addresses federated learning for simple bilevel optimization, where multiple clients aim to minimize a secondary global loss among the optimal inner-level solutions of a primary global loss. It introduces a universal regularization scheme (URS) that plugs a regularized objective $f_\eta(x)=h(x)+\eta f(x)$ into existing FL methods, yielding explicit error and distance guarantees that connect the regularized solution to the bilevel optimum. Building on URS, the authors develop R-FedAvg and R-SCAFFOLD with clear communication-complexity bounds under convex, strongly convex, and weak-sharp inner-outer structures, and extend to nonconvex outer-level settings via a two-loop scheme with provable rates. The theoretical results are complemented by extensive numerical experiments on over-parameterized sparse regression and image classification (EMNIST, CIFAR-10), validating the practical effectiveness and tuning guidance for $\eta$ and local steps $K$. Overall, this work provides the first provable, communication-efficient FL guarantees for simple bilevel optimization and demonstrates how URS can adapt standard FL methods to this challenging class of problems.
Abstract
We study a bilevel federated learning (FL) problem, where clients cooperatively seek to find among multiple optimal solutions of a primary distributed learning problem, a solution that minimizes a secondary distributed global loss function. This problem is motivated by model selection in over-parameterized machine learning, in that the outer-level objective is a suitably-defined regularizer and the inner-level objective is the training loss function. Despite recent progress in centralized settings, communication-efficient FL methods equipped with complexity guarantees for resolving this problem class are primarily absent. Motivated by this lacuna, we consider the setting where the inner-level objective is convex and the outer-level objective is either convex or strongly convex. We propose a universal regularized scheme and derive promising error bounds in terms of both the inner-level and outer-level loss functions. Leveraging this unifying theory, we then enable two existing FL methods to address the corresponding simple bilevel problem and derive novel communication complexity guarantees for each method. Additionally, we devise an FL method for addressing simple bilevel optimization problems with a nonconvex outer-level loss function. Through a two-loop scheme and by leveraging the universal theory, we derive new complexity bounds for the nonconvex setting. This appears to be the first time that federated simple bilevel optimization problems are provably addressed with guarantees. We validate the theoretical findings on EMNIST and CIFAR-10 datasets.