Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ColNet: Collaborative Optimization in Decentralized Federated Multi-task Learning Systems

Chao Feng, Nicolas Fazli Kohler, Zhi Wang, Weijie Niu, Alberto Huertas Celdran, Gerome Bovet, Burkhard Stiller

TL;DR

ColNet addresses the challenge of decentralized Federated Multi-Task Learning with task heterogeneity by partitioning models into a shared backbone and task-specific heads, forming task-coherent groups through adaptive clustering based on model and data sensitivity. Within-group backbones are averaged using FedAvg, while group leaders perform a hyper-conflict-averse (HCA) cross-group aggregation to mitigate gradient conflicts and promote cross-task knowledge sharing. The framework demonstrates consistent gains over baselines under label and task heterogeneity on CIFAR-10 and CelebA, and shows robustness to poisoning attacks, indicating strong privacy-preserving and resilient performance in distributed settings. The methods enable scalable, decentralized collaboration across heterogeneous tasks, with potential impact on privacy-preserving, multi-task optimization in real-world federated deployments.

Abstract

The integration of Federated Learning (FL) and Multi-Task Learning (MTL) has been explored to address client heterogeneity, with Federated Multi-Task Learning (FMTL) treating each client as a distinct task. However, most existing research focuses on data heterogeneity (e.g., addressing non-IID data) rather than task heterogeneity, where clients solve fundamentally different tasks. Additionally, much of the work relies on centralized settings with a server managing the federation, leaving the more challenging domain of decentralized FMTL largely unexplored. Thus, this work bridges this gap by proposing ColNet, a framework designed for heterogeneous tasks in decentralized federated environments. ColNet partitions models into a backbone and task-specific heads, and uses adaptive clustering based on model and data sensitivity to form task-coherent client groups. Backbones are averaged within groups, and group leaders perform hyper-conflict-averse cross-group aggregation. Across datasets and federations, ColNet outperforms competing schemes under label and task heterogeneity and shows robustness to poisoning attacks.

ColNet: Collaborative Optimization in Decentralized Federated Multi-task Learning Systems

TL;DR

ColNet addresses the challenge of decentralized Federated Multi-Task Learning with task heterogeneity by partitioning models into a shared backbone and task-specific heads, forming task-coherent groups through adaptive clustering based on model and data sensitivity. Within-group backbones are averaged using FedAvg, while group leaders perform a hyper-conflict-averse (HCA) cross-group aggregation to mitigate gradient conflicts and promote cross-task knowledge sharing. The framework demonstrates consistent gains over baselines under label and task heterogeneity on CIFAR-10 and CelebA, and shows robustness to poisoning attacks, indicating strong privacy-preserving and resilient performance in distributed settings. The methods enable scalable, decentralized collaboration across heterogeneous tasks, with potential impact on privacy-preserving, multi-task optimization in real-world federated deployments.

Abstract

The integration of Federated Learning (FL) and Multi-Task Learning (MTL) has been explored to address client heterogeneity, with Federated Multi-Task Learning (FMTL) treating each client as a distinct task. However, most existing research focuses on data heterogeneity (e.g., addressing non-IID data) rather than task heterogeneity, where clients solve fundamentally different tasks. Additionally, much of the work relies on centralized settings with a server managing the federation, leaving the more challenging domain of decentralized FMTL largely unexplored. Thus, this work bridges this gap by proposing ColNet, a framework designed for heterogeneous tasks in decentralized federated environments. ColNet partitions models into a backbone and task-specific heads, and uses adaptive clustering based on model and data sensitivity to form task-coherent client groups. Backbones are averaged within groups, and group leaders perform hyper-conflict-averse cross-group aggregation. Across datasets and federations, ColNet outperforms competing schemes under label and task heterogeneity and shows robustness to poisoning attacks.
Paper Structure (26 sections, 9 equations, 8 figures, 4 tables, 3 algorithms)

This paper contains 26 sections, 9 equations, 8 figures, 4 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. The ColNet Solution
  4. Problem Statement
  5. ColNet Architecture
  6. Learning Process
  7. Local Learning
  8. Model Exchanging
  9. Adaptive Clustering and Intra-Group Aggregation
  10. Cross-Group Aggregation
  11. Model Redistribution
  12. Deployment and Evaluation
  13. Scenarios, Datasets, and Models
  14. Federation Setup
  15. Impact of Aggregation Frequency
  16. ...and 11 more sections

Figures (8)

  • Figure 1: Overview of the ColNet Learning Process
  • Figure 2: Average Client Validation Loss in Each Epoch with Different Local Epoch Setups
  • Figure 3: Average Client Validation Loss in Each Epoch with Different Privatization Setups
  • Figure 4: Average Client Validation Loss in Each Epoch for Different Aggregation Schemes
  • Figure 5: Impact of Untargeted Label Flipping Attack on CIFAR-10 and CelebA Datasets
  • ...and 3 more figures