Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedCert: Federated Accuracy Certification

Minh Hieu Nguyen, Huu Tien Nguyen, Trung Thanh Nguyen, Manh Duong Nguyen, Trong Nghia Hoang, Truong Thao Nguyen, Phi Le Nguyen

TL;DR

This paper tackles the challenge of certifying the robustness of federated learning models when the server lacks access to a shared test set. It introduces FedCert, which approximates the global certified accuracy $c(\theta,S,r)$ by combining per-client certified accuracies and class distributions through convex optimization, while handling non-IID data via volume-based client grouping. Theoretical results establish linearity of certified accuracy across datasets and provide a bound on estimation error that depends on distribution mismatch $\delta$ and a dataset-dependent constant $Q$. Empirical evaluations on CIFAR-10 and CIFAR-100 demonstrate that the grouped-aggregation approach (GA) consistently reduces estimation error (RMSE and MAPE) compared to the baseline VW method across multiple FL algorithms and data distributions, with code available at the project repository. Overall, FedCert offers a practical, scalable method to assess the robustness of decentralized learning systems and supports more dependable FL deployments.

Abstract

Federated Learning (FL) has emerged as a powerful paradigm for training machine learning models in a decentralized manner, preserving data privacy by keeping local data on clients. However, evaluating the robustness of these models against data perturbations on clients remains a significant challenge. Previous studies have assessed the effectiveness of models in centralized training based on certified accuracy, which guarantees that a certain percentage of the model's predictions will remain correct even if the input data is perturbed. However, the challenge of extending these evaluations to FL remains unresolved due to the unknown client's local data. To tackle this challenge, this study proposed a method named FedCert to take the first step toward evaluating the robustness of FL systems. The proposed method is designed to approximate the certified accuracy of a global model based on the certified accuracy and class distribution of each client. Additionally, considering the Non-Independent and Identically Distributed (Non-IID) nature of data in real-world scenarios, we introduce the client grouping algorithm to ensure reliable certified accuracy during the aggregation step of the approximation algorithm. Through theoretical analysis, we demonstrate the effectiveness of FedCert in assessing the robustness and reliability of FL systems. Moreover, experimental results on the CIFAR-10 and CIFAR-100 datasets under various scenarios show that FedCert consistently reduces the estimation error compared to baseline methods. This study offers a solution for evaluating the robustness of FL systems and lays the groundwork for future research to enhance the dependability of decentralized learning. The source code is available at https://github.com/thanhhff/FedCert/.

FedCert: Federated Accuracy Certification

TL;DR

This paper tackles the challenge of certifying the robustness of federated learning models when the server lacks access to a shared test set. It introduces FedCert, which approximates the global certified accuracy by combining per-client certified accuracies and class distributions through convex optimization, while handling non-IID data via volume-based client grouping. Theoretical results establish linearity of certified accuracy across datasets and provide a bound on estimation error that depends on distribution mismatch and a dataset-dependent constant . Empirical evaluations on CIFAR-10 and CIFAR-100 demonstrate that the grouped-aggregation approach (GA) consistently reduces estimation error (RMSE and MAPE) compared to the baseline VW method across multiple FL algorithms and data distributions, with code available at the project repository. Overall, FedCert offers a practical, scalable method to assess the robustness of decentralized learning systems and supports more dependable FL deployments.

Abstract

Federated Learning (FL) has emerged as a powerful paradigm for training machine learning models in a decentralized manner, preserving data privacy by keeping local data on clients. However, evaluating the robustness of these models against data perturbations on clients remains a significant challenge. Previous studies have assessed the effectiveness of models in centralized training based on certified accuracy, which guarantees that a certain percentage of the model's predictions will remain correct even if the input data is perturbed. However, the challenge of extending these evaluations to FL remains unresolved due to the unknown client's local data. To tackle this challenge, this study proposed a method named FedCert to take the first step toward evaluating the robustness of FL systems. The proposed method is designed to approximate the certified accuracy of a global model based on the certified accuracy and class distribution of each client. Additionally, considering the Non-Independent and Identically Distributed (Non-IID) nature of data in real-world scenarios, we introduce the client grouping algorithm to ensure reliable certified accuracy during the aggregation step of the approximation algorithm. Through theoretical analysis, we demonstrate the effectiveness of FedCert in assessing the robustness and reliability of FL systems. Moreover, experimental results on the CIFAR-10 and CIFAR-100 datasets under various scenarios show that FedCert consistently reduces the estimation error compared to baseline methods. This study offers a solution for evaluating the robustness of FL systems and lays the groundwork for future research to enhance the dependability of decentralized learning. The source code is available at https://github.com/thanhhff/FedCert/.
Paper Structure (21 sections, 31 equations, 2 figures, 5 tables, 2 algorithms)

This paper contains 21 sections, 31 equations, 2 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Federated Learning
  4. Certified Accuracy
  5. Robustness in Federated Learning
  6. Methodology
  7. Theoretical Analysis
  8. Practical Algorithm
  9. Overview
  10. Local Accuracy Certification
  11. Volume-based Client Grouping
  12. Global Accuracy Certification
  13. Evaluation
  14. Experimental Settings
  15. Experimental Results
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Overview of FedCert. The clients validate the accuracy of the global model using their local datasets, the server consolidates small clients into a virtual large client, and then combines all the locally validated accuracies to estimate the final certified accuracy of the global model.
  • Figure 2: Performance under different desired data distributions (PS) and the test sample distributions of all clients (PD). (ResNet-18, CIFAR-10 dataset, Pareto, $\beta=2$, FedAvg).