FedNE: Surrogate-Assisted Federated Neighbor Embedding for Dimensionality Reduction

TL;DR

A novel approach that integrates the FedAvg framework with the contrastive NE technique, without any requirements of shareable data is introduced, and a surrogate loss function that each client learns and shares with each other is developed to address the lack of inter-client repulsion.

Abstract

Federated learning (FL) has rapidly evolved as a promising paradigm that enables collaborative model training across distributed participants without exchanging their local data. Despite its broad applications in fields such as computer vision, graph learning, and natural language processing, the development of a data projection model that can be effectively used to visualize data in the context of FL is crucial yet remains heavily under-explored. Neighbor embedding (NE) is an essential technique for visualizing complex high-dimensional data, but collaboratively learning a joint NE model is difficult. The key challenge lies in the objective function, as effective visualization algorithms like NE require computing loss functions among pairs of data. In this paper, we introduce \textsc{FedNE}, a novel approach that integrates the \textsc{FedAvg} framework with the contrastive NE technique, without any requirements of shareable data. To address the lack of inter-client repulsion which is crucial for the alignment in the global embedding space, we develop a surrogate loss function that each client learns and shares with each other. Additionally, we propose a data-mixing strategy to augment the local data, aiming to relax the problems of invisible neighbors and false neighbors constructed by the local $k$NN graphs. We conduct comprehensive experiments on both synthetic and real-world datasets. The results demonstrate that our \textsc{FedNE} can effectively preserve the neighborhood data structures and enhance the alignment in the global embedding space compared to several baseline methods.

Figures (10)

• Figure 1: An illustration of one round of FedNE. Besides the general steps in FedAvg: ① $\to$ ⑤ $\to$ ⑥, our local surrogate model training (②) can be smoothly integrated into the whole pipeline. Then, each client conducts its local training (⑤) using the augmented data and the surrogate models received from all the other clients (③ $\to$ ④).
• Figure 2: Toy examples for illustrating the major challenges in solving Federated NE. Color denotes the client identity, and different shapes represent the true categories of the data points (just for demonstration purposes). (a) Without repelling the dissimilar data from other clients, the projected data points from different clients may overlap with each other in the global embedding space. (b) Biased local $k$NN graphs may incorrectly connect distant data pairs as neighbors.
• Figure 3: Visualization of the resulting global test 2D embeddings on the Fashion-MNIST dataset under four FL setups of 20 clients ($M=20$).
• Figure 4: Experimental results on different step sizes in grid sampling for training the surrogate models. The experiments are conducted under the setting of $Dirichlet(0.1)$ on the MNIST dataset with 20 clients. In the main paper, the default step size is set to $0.3$. The results demonstrate that the performance of FedNE is stable when the step size is below 1.0.
• Figure 5: The quantitative evaluation on testing the surrogate function under five different retraining frequencies. The line chart shows the results of the MNIST test data with the FL setting of 20 clients and the Shards partition with two classes of data per client.