Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-supervised Cross-silo Federated Neural Architecture Search

Xinle Liang, Yang Liu, Jiahuan Luo, Yuanqin He, Tianjian Chen, Qiang Yang

TL;DR

Self-supervised Vertical Federated Neural Architecture Search (SS-VFNAS) is presented for automating FL where participants hold feature-partitioned data and is capable of generating high-performance and highly-transferable heterogeneous architectures even with insufficient overlapping samples, providing automation for those parties without deep learning expertise.

Abstract

Federated Learning (FL) provides both model performance and data privacy for machine learning tasks where samples or features are distributed among different parties. In the training process of FL, no party has a global view of data distributions or model architectures of other parties. Thus the manually-designed architectures may not be optimal. In the past, Neural Architecture Search (NAS) has been applied to FL to address this critical issue. However, existing Federated NAS approaches require prohibitive communication and computation effort, as well as the availability of high-quality labels. In this work, we present Self-supervised Vertical Federated Neural Architecture Search (SS-VFNAS) for automating FL where participants hold feature-partitioned data, a common cross-silo scenario called Vertical Federated Learning (VFL). In the proposed framework, each party first conducts NAS using self-supervised approach to find a local optimal architecture with its own data. Then, parties collaboratively improve the local optimal architecture in a VFL framework with supervision. We demonstrate experimentally that our approach has superior performance, communication efficiency and privacy compared to Federated NAS and is capable of generating high-performance and highly-transferable heterogeneous architectures even with insufficient overlapping samples, providing automation for those parties without deep learning expertise.

Self-supervised Cross-silo Federated Neural Architecture Search

TL;DR

Self-supervised Vertical Federated Neural Architecture Search (SS-VFNAS) is presented for automating FL where participants hold feature-partitioned data and is capable of generating high-performance and highly-transferable heterogeneous architectures even with insufficient overlapping samples, providing automation for those parties without deep learning expertise.

Abstract

Federated Learning (FL) provides both model performance and data privacy for machine learning tasks where samples or features are distributed among different parties. In the training process of FL, no party has a global view of data distributions or model architectures of other parties. Thus the manually-designed architectures may not be optimal. In the past, Neural Architecture Search (NAS) has been applied to FL to address this critical issue. However, existing Federated NAS approaches require prohibitive communication and computation effort, as well as the availability of high-quality labels. In this work, we present Self-supervised Vertical Federated Neural Architecture Search (SS-VFNAS) for automating FL where participants hold feature-partitioned data, a common cross-silo scenario called Vertical Federated Learning (VFL). In the proposed framework, each party first conducts NAS using self-supervised approach to find a local optimal architecture with its own data. Then, parties collaboratively improve the local optimal architecture in a VFL framework with supervision. We demonstrate experimentally that our approach has superior performance, communication efficiency and privacy compared to Federated NAS and is capable of generating high-performance and highly-transferable heterogeneous architectures even with insufficient overlapping samples, providing automation for those parties without deep learning expertise.

Paper Structure

This paper contains 23 sections, 2 theorems, 14 equations, 9 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Vertical Federated Learning
  4. Federated NAS
  5. Self-Supervised NAS
  6. Problem Formulation
  7. Methods
  8. Vertical Federated Deep Learning
  9. Naive VFNAS Methods
  10. Privacy Preservation by Differential Privacy
  11. Communication Efficiency and Privacy Improvement by Self-Supervised Learning
  12. Experiments
  13. Experimental Settings
  14. Dataset
  15. Architecture Search Space
  16. ...and 8 more sections

Key Result

Theorem 1

As shown in Eqn. dp_formulation, at each step a Gaussian mechanism $\mathcal{M}^{\mathcal{N}_j}$ adds a Gaussian random noise to the output of neural network $\mathcal{N}_j$ from party $j$. This guarantees ($\varepsilon_1, \delta_1$)-differential privacy for each step, if we choose $\sigma_1$ to be

Figures (9)

  • Figure 1: Cross-hospital AD diagnosis VFL system. Net A and B are separately maintained by different hospitals to extract complementary information of PET and MRI data.
  • Figure 2: Vertical Federated Deep Learning. Different parties maintain their own network models, which is updated by the exchanges of the intermediate outputs and their corresponding gradients.
  • Figure 3: The process for reforging ModelNet40 for VFL classification benchmark. Firstly, we generate multi-view images based on the approaches in Su2015Multi. Then we distribute the images evenly to different parties. Each VFL sample is generated by taking one single-view image from each party in sequence.
  • Figure 4: Test Accuracy VS Model Size.
  • Figure 5: Optimal network architecture searched by SS-VFNAS$^{1}$-M with parties 6 and 5 in FedModelNet40.
  • ...and 4 more figures

Theorems & Definitions (3)

  • Definition 1
  • Theorem 1
  • Corollary 1