Federated Multi-Task Clustering

TL;DR

This paper addresses the challenge of privacy-preserving clustering across heterogeneous clients by proposing Federated Multi-Task Clustering (FMTC), which learns client-specific projections and spectral embeddings while explicitly modeling inter-client relationships via a low-rank tensor regularizer. The method comprises a client-side module that jointly learns $\mathbf{F}_t$ and $\mathbf{W}_t$ and a server-side tensorial correlation module that enforces cross-client consistency on a third-order tensor $\bm{\mathcal{W}}$, optimized with a privacy-preserving ADMM framework. Empirical results on seven real-world datasets show FMTC consistently outperforms baselines, including federated clustering methods and centralized multi-task clustering, particularly in out-of-sample generalization. The work advances privacy-aware, personalized clustering by enabling robust, transferable representations across non-IID clients while preserving local structure.

Abstract

Spectral clustering has emerged as one of the most effective clustering algorithms due to its superior performance. However, most existing models are designed for centralized settings, rendering them inapplicable in modern decentralized environments. Moreover, current federated learning approaches often suffer from poor generalization performance due to reliance on unreliable pseudo-labels, and fail to capture the latent correlations amongst heterogeneous clients. To tackle these limitations, this paper proposes a novel framework named Federated Multi-Task Clustering (i.e.,FMTC), which intends to learn personalized clustering models for heterogeneous clients while collaboratively leveraging their shared underlying structure in a privacy-preserving manner. More specifically, the FMTC framework is composed of two main components: client-side personalized clustering module, which learns a parameterized mapping model to support robust out-of-sample inference, bypassing the need for unreliable pseudo-labels; and server-side tensorial correlation module, which explicitly captures the shared knowledge across all clients. This is achieved by organizing all client models into a unified tensor and applying a low-rank regularization to discover their common subspace. To solve this joint optimization problem, we derive an efficient, privacy-preserving distributed algorithm based on the Alternating Direction Method of Multipliers, which decomposes the global problem into parallel local updates on clients and an aggregation step on the server. To the end, several extensive experiments on multiple real-world datasets demonstrate that our proposed FMTC framework significantly outperforms various baseline and state-of-the-art federated clustering algorithms.

Figures (6)

• Figure 1: Motivation of our federated multi-task clustering model, where clients in each round share local model updates and the server aggregates these into a model tensor with multi-task constraint, distilling shared structure which is then broadcast back to refine each personalized model.
• Figure 2: The architecture of our Federated Multi-Task Clustering (FMTC) framework, where the process begins with heterogeneous data samples $\{X_t\}$ from diverse clients. At the client level, each client data is processed through Local Spectral Embedding, which is optimized according to our local objective function. This process generates a client-specific Personalized Feature Space and learns a corresponding Personalized Clustering Model $W_t$. At the server level, all personalized clustering models $\{W_t\}$ are collected and stacked into a Knowledge Fusion Tensor $\bm{\mathcal{W}}$. The Tensorial Correlation Module is then applied to this tensor to distill a Global Consensus $\bm{\mathcal{Z}}$, representing the shared knowledge across all tasks. This consensus is further used to obtain the Global Refined Clustering Model. Finally, the Collaborative Guidance from the server provides regularization back to the local spectral embedding, completing the federated optimization loop.
• Figure 3: t-SNE visualization of the WebKB, 20Newsgroups, Reuters, and Keck datasets. Each color represents a distinct class. The plots highlight the significant heterogeneity and complex, non-linearly separable structures within and across tasks (clients), motivating the need for a personalized and collaborative clustering approach like FMTC.
• Figure 4: Convergence analysis of our proposed FMTC framework on three datasets. For each dataset, we plot the overall objective function value and the average clustering ACC (%) versus the number of iterations.
• Figure 5: Parameter sensitivity analysis of FMTC on the WebKB4 dataset. The figure shows the ACC (%) as a function of the collaboration weight $\beta$ and the local fidelity weight $\alpha$. Each row, corresponding to a specific $\alpha$, is rendered in a unique color.

Theorems & Definitions (1)

• proof