How Does the Smoothness Approximation Method Facilitate Generalization for Federated Adversarial Learning?

TL;DR

The paper tackles the generalization gap in Federated Adversarial Learning (FAL) caused by non-smooth adversarial losses. It introduces three smoothness-approximation methods—Surrogate Smoothness Approximation (SSA), Randomized Smoothness Approximation (RSA), and Over-Parameterized Smoothness Approximation (OPSA)—to study the stability-based generalization bounds of Vanilla FAL (VFAL) and Slack FAL (SFAL). Theoretical results show RSA consistently delivers the best generalization performance, while SFAL further improves generalization under data heterogeneity through an $\alpha$-slack aggregation mechanism; OPSA highlights a trade-off with network width. These insights guide the design of more efficient, robust FAL algorithms and suggest new metrics and dynamic aggregation rules to mitigate heterogeneity. The empirical evaluations on non-IID datasets corroborate the theory, demonstrating RSA’s superiority and SFAL’s advantage over VFAL in heterogeneous settings.

Abstract

Federated Adversarial Learning (FAL) is a robust framework for resisting adversarial attacks on federated learning. Although some FAL studies have developed efficient algorithms, they primarily focus on convergence performance and overlook generalization. Generalization is crucial for evaluating algorithm performance on unseen data. However, generalization analysis is more challenging due to non-smooth adversarial loss functions. A common approach to addressing this issue is to leverage smoothness approximation. In this paper, we develop algorithm stability measures to evaluate the generalization performance of two popular FAL algorithms: \textit{Vanilla FAL (VFAL)} and {\it Slack FAL (SFAL)}, using three different smooth approximation methods: 1) \textit{Surrogate Smoothness Approximation (SSA)}, (2) \textit{Randomized Smoothness Approximation (RSA)}, and (3) \textit{Over-Parameterized Smoothness Approximation (OPSA)}. Based on our in-depth analysis, we answer the question of how to properly set the smoothness approximation method to mitigate generalization error in FAL. Moreover, we identify RSA as the most effective method for reducing generalization error. In highly data-heterogeneous scenarios, we also recommend employing SFAL to mitigate the deterioration of generalization performance caused by heterogeneity. Based on our theoretical results, we provide insights to help develop more efficient FAL algorithms, such as designing new metrics and dynamic aggregation rules to mitigate heterogeneity.

Figures (7)

• Figure 1: Generalization Gap of the attack strength $\rho$ on SVHN. ($m=40,a=2.0$)
• Figure 2: Generalization Gap of the skew parameter $a$ on SVHN. ($m=40,\rho=1.0$)
• Figure 3: Generalization Gap of the number of client $m$ on SVHN. ($a=1.0,\rho=1.0$)
• Figure 4: Generalization Gap of the number of noise $Q$ on SVHN. ($m=40,a=2.0,\rho=2.0$)
• Figure 5: Generalization Gap of the attack strength $\rho$ on CIFAR10. ($m=40,a=2.0$)

Theorems & Definitions (50)

• Definition 1: Neighboring Datasets
• Definition 2: On-Average Stability for FAL
• Lemma 1
• Theorem 1
• Definition 3
• Theorem 2
• Theorem 3
• Theorem 4
• Theorem 5
• Theorem 6