Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Robustness of Distributed Machine Learning against Transfer Attacks

Sébastien Andreina, Pascal Zimmer, Ghassan Karame

TL;DR

This work addresses the robustness of distributed machine learning when both learning and inference are distributed across heterogeneous workers. It analyzes transfer-based attacks and proposes a hybrid distributed paradigm with diverse training data, architectures, optimizers, and schedulers to reduce attack transferability, supported by theory and extensive CIFAR10/FashionMNIST experiments. Key findings show substantial improvements in robust accuracy (RA) across CW/SAM/CSE attacks, with RA gains up to around $40\%$ and minimal losses in clean accuracy, particularly as the number of nodes $N$ increases and hyperparameters are independently tuned. The results suggest that increasing node count and hyperparameter diversity can meaningfully enhance robustness in distributed ML, while diversity in architecture/optimizer/scheduler plays a lesser role, highlighting practical directions for robust, privacy-aware distributed deployments.

Abstract

Although distributed machine learning (distributed ML) is gaining considerable attention in the community, prior works have independently looked at instances of distributed ML in either the training or the inference phase. No prior work has examined the combined robustness stemming from distributing both the learning and the inference process. In this work, we explore, for the first time, the robustness of distributed ML models that are fully heterogeneous in training data, architecture, scheduler, optimizer, and other model parameters. Supported by theory and extensive experimental validation using CIFAR10 and FashionMNIST, we show that such properly distributed ML instantiations achieve across-the-board improvements in accuracy-robustness tradeoffs against state-of-the-art transfer-based attacks that could otherwise not be realized by current ensemble or federated learning instantiations. For instance, our experiments on CIFAR10 show that for the Common Weakness attack, one of the most powerful state-of-the-art transfer-based attacks, our method improves robust accuracy by up to 40%, with a minimal impact on clean task accuracy.

On the Robustness of Distributed Machine Learning against Transfer Attacks

TL;DR

This work addresses the robustness of distributed machine learning when both learning and inference are distributed across heterogeneous workers. It analyzes transfer-based attacks and proposes a hybrid distributed paradigm with diverse training data, architectures, optimizers, and schedulers to reduce attack transferability, supported by theory and extensive CIFAR10/FashionMNIST experiments. Key findings show substantial improvements in robust accuracy (RA) across CW/SAM/CSE attacks, with RA gains up to around and minimal losses in clean accuracy, particularly as the number of nodes increases and hyperparameters are independently tuned. The results suggest that increasing node count and hyperparameter diversity can meaningfully enhance robustness in distributed ML, while diversity in architecture/optimizer/scheduler plays a lesser role, highlighting practical directions for robust, privacy-aware distributed deployments.

Abstract

Although distributed machine learning (distributed ML) is gaining considerable attention in the community, prior works have independently looked at instances of distributed ML in either the training or the inference phase. No prior work has examined the combined robustness stemming from distributing both the learning and the inference process. In this work, we explore, for the first time, the robustness of distributed ML models that are fully heterogeneous in training data, architecture, scheduler, optimizer, and other model parameters. Supported by theory and extensive experimental validation using CIFAR10 and FashionMNIST, we show that such properly distributed ML instantiations achieve across-the-board improvements in accuracy-robustness tradeoffs against state-of-the-art transfer-based attacks that could otherwise not be realized by current ensemble or federated learning instantiations. For instance, our experiments on CIFAR10 show that for the Common Weakness attack, one of the most powerful state-of-the-art transfer-based attacks, our method improves robust accuracy by up to 40%, with a minimal impact on clean task accuracy.

Paper Structure

This paper contains 22 sections, 2 theorems, 12 equations, 2 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. ML Paradigms
  4. Transfer-based attacks
  5. Related Work
  6. Methodology
  7. Preliminaries and Notations
  8. Main Intuition
  9. Experimental Approach
  10. Training parameters
  11. Parameter tuning
  12. Impact of Parameter Diversity
  13. Experiments
  14. Experimental Setup
  15. Selection of the Attacks
  16. ...and 7 more sections

Key Result

Proposition 1

A model $f$ with major parameter configuration $\mathcal{P}$ and hyperparameter configuration $\mathcal{H}$ is optimized to model parameters $\theta$ during training. Heterogeneity, i.e., a change in these parameter configurations, $\hat{\mathcal{P}}, \hat{\mathcal{H}}$, results in a model $\hat{f}$

Figures (2)

  • Figure 1: Pareto frontier of all accuracy-robustness tradeoffs of distributed instantiations (D$_\mathcal{P}\xspace$) compared to the baseline ensemble (ENS).
  • Figure 2: Heatmap of the similarity of individual models within our different scenarios with the surrogate models.

Theorems & Definitions (2)

  • Proposition 1
  • Proposition 2