ProxyFL: A Proxy-Guided Framework for Federated Semi-Supervised Learning

TL;DR

A proxy-guided framework called ProxyFL that focuses on simultaneously mitigating external and internal heterogeneity via a unified proxy, and considers the learnable weights of classifier as proxy to simulate the category distribution both locally and globally.

Abstract

Federated Semi-Supervised Learning (FSSL) aims to collaboratively train a global model across clients by leveraging partially-annotated local data in a privacy-preserving manner. In FSSL, data heterogeneity is a challenging issue, which exists both across clients and within clients. External heterogeneity refers to the data distribution discrepancy across different clients, while internal heterogeneity represents the mismatch between labeled and unlabeled data within clients. Most FSSL methods typically design fixed or dynamic parameter aggregation strategies to collect client knowledge on the server (external) and / or filter out low-confidence unlabeled samples to reduce mistakes in local client (internal). But, the former is hard to precisely fit the ideal global distribution via direct weights, and the latter results in fewer data participation into FL training. To this end, we propose a proxy-guided framework called ProxyFL that focuses on simultaneously mitigating external and internal heterogeneity via a unified proxy. I.e., we consider the learnable weights of classifier as proxy to simulate the category distribution both locally and globally. For external, we explicitly optimize global proxy against outliers instead of direct weights; for internal, we re-include the discarded samples into training by a positive-negative proxy pool to mitigate the impact of potentially-incorrect pseudo-labels. Insight experiments & theoretical analysis show our significant performance and convergence in FSSL.

Figures (7)

• Figure 1: (a) Illustration of centralized learning and averaging-based federated learning. (b) Differences of test accuracy and pseudo-labeling accuracy under varying levels of heterogeneity (smaller $\alpha$ indicates greater heterogeneity). During each communication round, all clients are trained based on FedSGD mcmahan2017fedavgw/, w/o low-confidence samples and our method for one local epoch.
• Figure 2: (a) t-SNE visualization of classifier weights from clients (circle) during the initial round. Different colors means different categories. Pentagram denotes the simple average of weight parameters by category. (b) Number of excluded unlabeled samples under different levels of heterogeneity. (c) Test accuracy curves of 'SGD-FSSL' w/o and w/ low-confidence samples under different ratios of correct labels. These samples could improve performance as the correctly-labeled number increases.
• Figure 3: Overview of ProxyFL for Global Proxy Tuning in server-side & Indecisive-Categories Proxy Learning in local client.
• Figure 4: (a-b) Examples of the indecisive-categories. (c) Illustration of our Indecisive-Categories Proxy Learning.
• Figure 5: (a) Distribution of labeled and unlabeled data across clients under $\alpha=0.1$ taking CIFAR-10 as an example. (b) Convergence curves of ProxyFL and other baselines on CIFAR-100 with $\alpha = 0.1$. (c) Distribution of global category proxies before-and after-tuning visualized in a t-SNE plot.

Theorems & Definitions (5)

• Remark 1
• Definition 1
• Definition 2
• Theorem 1
• Theorem 2