Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated Incremental Subgradient Method for Convex Bilevel Optimization Problems

Sudkobfa Boontawee, Mootta Prangprakhon, Nimit Nimana

TL;DR

This work tackles convex bilevel optimization in a federated setting, where the outer objective $H(x)$ is strongly convex and the inner objective is a finite-sum across clients. It introduces the Federated Incremental Subgradient Method (FISM), which blends iterative regularization with a federated update protocol, sharing only a subgradient $\mathcal{H}_k$ of $H$ to preserve privacy and reduce local computation. The authors prove convergence of the iterates to the unique outer-optimal $x_H^*$ and establish a near $1/K^{0.5}$ convergence rate for the inner-level objective value, under standard assumptions and suitably chosen step sizes. Empirical results on binary MNIST classification and a location problem demonstrate that FISM, especially with more groups $S$, achieves faster convergence and better accuracy than the IR-IG baseline, highlighting the method’s scalability and privacy benefits in distributed optimization settings. The work also discusses practical considerations and future directions, including asynchronous communication and extensions to nonconvex problems.

Abstract

In this letter, we consider a bilevel optimization problem in which the outer-level objective function is strongly convex, whereas the inner-level problem consists of a finite sum of convex functions. Bilevel optimization problems arise in situations where the inner-level problem does not have a unique solution. This has led to the idea of introducing an outer-level objective function to select a solution with the specific desired properties. We propose an iterative method that combines an incremental algorithm with a broadcast algorithm, both based on the principles of federated learning. Under appropriate assumptions, we establish the convergence results of the proposed algorithm. To demonstrate its performance, we present two numerical examples related to binary classification and a location problem.

Federated Incremental Subgradient Method for Convex Bilevel Optimization Problems

TL;DR

This work tackles convex bilevel optimization in a federated setting, where the outer objective is strongly convex and the inner objective is a finite-sum across clients. It introduces the Federated Incremental Subgradient Method (FISM), which blends iterative regularization with a federated update protocol, sharing only a subgradient of to preserve privacy and reduce local computation. The authors prove convergence of the iterates to the unique outer-optimal and establish a near convergence rate for the inner-level objective value, under standard assumptions and suitably chosen step sizes. Empirical results on binary MNIST classification and a location problem demonstrate that FISM, especially with more groups , achieves faster convergence and better accuracy than the IR-IG baseline, highlighting the method’s scalability and privacy benefits in distributed optimization settings. The work also discusses practical considerations and future directions, including asynchronous communication and extensions to nonconvex problems.

Abstract

In this letter, we consider a bilevel optimization problem in which the outer-level objective function is strongly convex, whereas the inner-level problem consists of a finite sum of convex functions. Bilevel optimization problems arise in situations where the inner-level problem does not have a unique solution. This has led to the idea of introducing an outer-level objective function to select a solution with the specific desired properties. We propose an iterative method that combines an incremental algorithm with a broadcast algorithm, both based on the principles of federated learning. Under appropriate assumptions, we establish the convergence results of the proposed algorithm. To demonstrate its performance, we present two numerical examples related to binary classification and a location problem.
Paper Structure (8 sections, 5 theorems, 26 equations, 3 figures, 1 table, 1 algorithm)

This paper contains 8 sections, 5 theorems, 26 equations, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem and Existing Methods
  3. Contributions
  4. Federated Incremental Subgradient Method
  5. Numerical Experiments
  6. Binary Image Classification
  7. Location Problem
  8. Concluding Remarks

Key Result

Lemma 4

YNS17 Let Assumption ass1 hold, and let $\{ \gamma_k \}_{k=1}^\infty$ be a positive sequence. Let $x_{\lambda_k}^*$ be the minimizer of Problem $(P_{\lambda_k})$ for all $k \geq 1$. Then,

Figures (3)

  • Figure 1: Workflow of FISM
  • Figure 2: Comparison of total number of observations $m$ and computational running time for FISM and IR-IG.
  • Figure 3: Comparison of performance for FISM and IR-IG

Theorems & Definitions (12)

  • Remark 2
  • Remark 3
  • Lemma 4
  • Lemma 5
  • proof
  • Remark 7
  • Lemma 8
  • Theorem 9
  • proof
  • Remark 10
  • ...and 2 more