Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FOOGD: Federated Collaboration for Both Out-of-distribution Generalization and Detection

Xinting Liao, Weiming Liu, Pengyang Zhou, Fengyuan Yu, Jiahe Xu, Jun Wang, Wenjie Wang, Chaochao Chen, Xiaolin Zheng

TL;DR

This work proposes FOOGD, a method that estimates the probability density of each client and obtains reliable global distribution as guidance for the subsequent FL process, and significantly enjoys three main advantages: reliably estimating non-normalized decentralized distributions, detecting semantic shift data via score values, and generalizing to covariate-shift data by regularizing feature extractor.

Abstract

Federated learning (FL) is a promising machine learning paradigm that collaborates with client models to capture global knowledge. However, deploying FL models in real-world scenarios remains unreliable due to the coexistence of in-distribution data and unexpected out-of-distribution (OOD) data, such as covariate-shift and semantic-shift data. Current FL researches typically address either covariate-shift data through OOD generalization or semantic-shift data via OOD detection, overlooking the simultaneous occurrence of various OOD shifts. In this work, we propose FOOGD, a method that estimates the probability density of each client and obtains reliable global distribution as guidance for the subsequent FL process. Firstly, SM3D in FOOGD estimates score model for arbitrary distributions without prior constraints, and detects semantic-shift data powerfully. Then SAG in FOOGD provides invariant yet diverse knowledge for both local covariate-shift generalization and client performance generalization. In empirical validations, FOOGD significantly enjoys three main advantages: (1) reliably estimating non-normalized decentralized distributions, (2) detecting semantic shift data via score values, and (3) generalizing to covariate-shift data by regularizing feature extractor. The prejoct is open in https://github.com/XeniaLLL/FOOGD-main.git.

FOOGD: Federated Collaboration for Both Out-of-distribution Generalization and Detection

TL;DR

This work proposes FOOGD, a method that estimates the probability density of each client and obtains reliable global distribution as guidance for the subsequent FL process, and significantly enjoys three main advantages: reliably estimating non-normalized decentralized distributions, detecting semantic shift data via score values, and generalizing to covariate-shift data by regularizing feature extractor.

Abstract

Federated learning (FL) is a promising machine learning paradigm that collaborates with client models to capture global knowledge. However, deploying FL models in real-world scenarios remains unreliable due to the coexistence of in-distribution data and unexpected out-of-distribution (OOD) data, such as covariate-shift and semantic-shift data. Current FL researches typically address either covariate-shift data through OOD generalization or semantic-shift data via OOD detection, overlooking the simultaneous occurrence of various OOD shifts. In this work, we propose FOOGD, a method that estimates the probability density of each client and obtains reliable global distribution as guidance for the subsequent FL process. Firstly, SM3D in FOOGD estimates score model for arbitrary distributions without prior constraints, and detects semantic-shift data powerfully. Then SAG in FOOGD provides invariant yet diverse knowledge for both local covariate-shift generalization and client performance generalization. In empirical validations, FOOGD significantly enjoys three main advantages: (1) reliably estimating non-normalized decentralized distributions, (2) detecting semantic shift data via score values, and (3) generalizing to covariate-shift data by regularizing feature extractor. The prejoct is open in https://github.com/XeniaLLL/FOOGD-main.git.

Paper Structure

This paper contains 36 sections, 6 theorems, 31 equations, 13 figures, 15 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. OOD Detection
  4. OOD Generalization
  5. Federated Learning with Wild Data
  6. Methodology
  7. Problem Setting
  8. S$\text{M}^3$D: Estimating Score Model for Detection
  9. SAG: Enhancing Feature Extractor for Gerneralization
  10. Theoretical Discussion
  11. Experiments
  12. Experimental Setups
  13. Experimental Results
  14. Conclusion and Future Work
  15. Algorithms
  16. ...and 21 more sections

Key Result

Theorem 4.1

Assume the original $\operatorname{MMD}(\boldsymbol{Z}, \boldsymbol{Z}_{\text{gen}})\leq C$ for randomly initialized score model $s_{\boldsymbol{\theta}}(\boldsymbol{z})$ in Eq. eq:mmd, the score model achieves optimum and MMD decreases. By Lemma supp_lemma:dsm, we can obtain the final error bound o where $C$ is the upper bound of the MMD, $B$ is batch size, and $|\mathcal{D}|$ is the data amount.

Figures (13)

  • Figure 1: Motivation of FOOGD. The distributions of two clients are non-IID, and we seek to estimate the global distribution among decentralized data.
  • Figure 2: Framework of FOOGD. For each client, we have main task feature extractor, a S$\text{M}^3$D module estimates score model (Eq. \ref{['eq:smd']}) for detection, and a SAG module regularizes feature extractor for enhancing generalization. The server aggregates models and obtains global distribution.
  • Figure 3: Motivation of S$\text{M}^3$D. Red points are sampled from target data distribution, and the blue points are generated by LDS in Eq. \ref{['eq:mcmc']}.
  • Figure 4: T-SNE visualizations of FedAvg and FedRod with FOOGD.
  • Figure 5: Detection score distribution of FL methods on Cifar10 ($\alpha=5.0$).
  • ...and 8 more figures

Theorems & Definitions (11)

  • Theorem 4.1: Error Bound of Decentralized Score Matching via S$\text{M}^3$D
  • Lemma B.1: Error Bound of Decentralized Score Matching
  • proof
  • Theorem B.2: Error Bound of Decentralized Score Matching via S$\text{M}^3$D
  • Lemma B.3: Stein identity
  • proof
  • Definition B.4: Stein Discrepancy
  • Lemma B.5
  • proof
  • Lemma B.10: Bound of Client Model Divergence
  • ...and 1 more