Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Smoothing ADMM for Non-convex and Non-smooth Hierarchical Federated Learning

Reza Mirzaeifard, Stefan Werner

TL;DR

The paper addresses optimization in hierarchical federated learning with non-convex and non-smooth objectives under client and cluster heterogeneity. It introduces Hierarchical Federated Smoothing ADMM (HFSAD), which blends smoothing upper-bounds with ADMM and auxiliary variables to enable asynchronous, multi-update distributed optimization across clusters and a global root. Regularizers are decomposed per cluster to support both consensus via total variation and personalization via non-convex penalties such as SCAD and MCP, while preserving proximal-friendly structure for efficient updates. Empirical results on SCAD-penalized robust phase retrieval show that HFSAD achieves faster convergence and higher accuracy than centralized baselines, highlighting its robustness and practicality for large-scale, heterogeneous FL scenarios.

Abstract

This paper presents a hierarchical federated learning (FL) framework that extends the alternating direction method of multipliers (ADMM) with smoothing techniques, tailored for non-convex and non-smooth objectives. Unlike traditional hierarchical FL methods, our approach supports asynchronous updates and multiple updates per iteration, enhancing adaptability to heterogeneous data and system settings. Additionally, we introduce a flexible mechanism to leverage diverse regularization functions at each layer, allowing customization to the specific prior information within each cluster and accommodating (possibly) non-smooth penalty objectives. Depending on the learning goal, the framework supports both consensus and personalization: the total variation norm can be used to enforce consensus across layers, while non-convex penalties such as minimax concave penalty (MCP) or smoothly clipped absolute deviation (SCAD) enable personalized learning. Experimental results demonstrate the superior convergence rates and accuracy of our method compared to conventional approaches, underscoring its robustness and versatility for a wide range of FL scenarios.

Smoothing ADMM for Non-convex and Non-smooth Hierarchical Federated Learning

TL;DR

The paper addresses optimization in hierarchical federated learning with non-convex and non-smooth objectives under client and cluster heterogeneity. It introduces Hierarchical Federated Smoothing ADMM (HFSAD), which blends smoothing upper-bounds with ADMM and auxiliary variables to enable asynchronous, multi-update distributed optimization across clusters and a global root. Regularizers are decomposed per cluster to support both consensus via total variation and personalization via non-convex penalties such as SCAD and MCP, while preserving proximal-friendly structure for efficient updates. Empirical results on SCAD-penalized robust phase retrieval show that HFSAD achieves faster convergence and higher accuracy than centralized baselines, highlighting its robustness and practicality for large-scale, heterogeneous FL scenarios.

Abstract

This paper presents a hierarchical federated learning (FL) framework that extends the alternating direction method of multipliers (ADMM) with smoothing techniques, tailored for non-convex and non-smooth objectives. Unlike traditional hierarchical FL methods, our approach supports asynchronous updates and multiple updates per iteration, enhancing adaptability to heterogeneous data and system settings. Additionally, we introduce a flexible mechanism to leverage diverse regularization functions at each layer, allowing customization to the specific prior information within each cluster and accommodating (possibly) non-smooth penalty objectives. Depending on the learning goal, the framework supports both consensus and personalization: the total variation norm can be used to enforce consensus across layers, while non-convex penalties such as minimax concave penalty (MCP) or smoothly clipped absolute deviation (SCAD) enable personalized learning. Experimental results demonstrate the superior convergence rates and accuracy of our method compared to conventional approaches, underscoring its robustness and versatility for a wide range of FL scenarios.

Paper Structure

This paper contains 5 sections, 1 theorem, 17 equations, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Hierarchical Federated Smoothing ADMM
  4. Simulation Results
  5. Conclusion

Key Result

Theorem 1

Assume that the each function $f_{lj}(\cdot)$, $R_w^l(\cdot), \forall l \in [L]\cup0$ has bounded gradients indicated by $\nu_f$ and $\nu_r$ respectively, parameters $\sigma_l^j, \mu_l^j, \sigma_l, \mu_l$ are updated according to eq:up:sm, $\alpha c \geq \sqrt{20}\nu_f$ and $\beta d \geq \sqrt{20}(

Figures (3)

  • Figure 1: Relative error comparison of HFSAD and Sub
  • Figure 2: Relative error comparison across different update steps
  • Figure 3: Relative error comparison across different collaboration probability

Theorems & Definitions (1)

  • Theorem 1