Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CAFEDistill: Learning Personalized and Dynamic Models through Federated Early-Exit Network Distillation

Boyi Liu, Zimu Zhou, Yongxin Tong

TL;DR

CAFEDistill extends Personalized Federated Learning to early-exit networks by addressing two core conflicts: client-wise heterogeneity and depth-wise interference. It introduces a progressive, depth-prioritized coordination strategy and a client-decoupled cross-client knowledge distillation, enabling effective personalized transfer with minimal communication overhead. Empirical results across four datasets show CAFEDistill consistently outperforms state-of-the-art PFL and GFL-EE baselines in accuracy while reducing inference costs by 30.79% to 46.86%. This work advances practical, dynamic, decentralized inference for heterogeneous IoT data and suggests directions for resource-aware and multimodal extensions.

Abstract

Personalized Federated Learning (PFL) enables collaboratively model training on decentralized, heterogeneous data while tailoring them to each client's unique distribution. However, existing PFL methods produce static models with a fixed tradeoff between accuracy and efficiency, limiting their applicability in environments where inference requirements vary with contexts and resource availability. Early-exit networks (EENs) offer adaptive inference by attaching intermediate classifiers. Yet integrating them into PFL is challenging due to client-wise heterogeneity and depth-wise interference arising from conflicting exit objectives. Prior studies fail to resolve both conflicts simultaneously, leading to suboptimal performance. In this paper, we propose CAFEDistill, a Conflict-Aware Federated Exit Distillation framework that jointly addresses these conflicts and extends PFL to early-exit networks. Through a progressive, depth-prioritized student coordination mechanism, CAFEDistill mitigates interference among shallow and deep exits while allowing effective personalized knowledge transfer across clients. Furthermore, it reduces communication overhead via a client-decoupled formulation. Extensive evaluations show that CAFEDistill outperforms the state-of-the-arts, achieving higher accuracy and reducing inference costs by 30.79%-46.86%.

CAFEDistill: Learning Personalized and Dynamic Models through Federated Early-Exit Network Distillation

TL;DR

CAFEDistill extends Personalized Federated Learning to early-exit networks by addressing two core conflicts: client-wise heterogeneity and depth-wise interference. It introduces a progressive, depth-prioritized coordination strategy and a client-decoupled cross-client knowledge distillation, enabling effective personalized transfer with minimal communication overhead. Empirical results across four datasets show CAFEDistill consistently outperforms state-of-the-art PFL and GFL-EE baselines in accuracy while reducing inference costs by 30.79% to 46.86%. This work advances practical, dynamic, decentralized inference for heterogeneous IoT data and suggests directions for resource-aware and multimodal extensions.

Abstract

Personalized Federated Learning (PFL) enables collaboratively model training on decentralized, heterogeneous data while tailoring them to each client's unique distribution. However, existing PFL methods produce static models with a fixed tradeoff between accuracy and efficiency, limiting their applicability in environments where inference requirements vary with contexts and resource availability. Early-exit networks (EENs) offer adaptive inference by attaching intermediate classifiers. Yet integrating them into PFL is challenging due to client-wise heterogeneity and depth-wise interference arising from conflicting exit objectives. Prior studies fail to resolve both conflicts simultaneously, leading to suboptimal performance. In this paper, we propose CAFEDistill, a Conflict-Aware Federated Exit Distillation framework that jointly addresses these conflicts and extends PFL to early-exit networks. Through a progressive, depth-prioritized student coordination mechanism, CAFEDistill mitigates interference among shallow and deep exits while allowing effective personalized knowledge transfer across clients. Furthermore, it reduces communication overhead via a client-decoupled formulation. Extensive evaluations show that CAFEDistill outperforms the state-of-the-arts, achieving higher accuracy and reducing inference costs by 30.79%-46.86%.
Paper Structure (32 sections, 2 theorems, 12 equations, 10 figures, 6 tables, 2 algorithms)

This paper contains 32 sections, 2 theorems, 12 equations, 10 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Statement
  4. Method
  5. CAFEDistill Overview
  6. Conflict-Aware Student Selection
  7. Cross-Client Teacher Matching
  8. Decoupled Cross-Client Distillation
  9. Putting It Together
  10. Experiments
  11. Experimental Setup
  12. Main Results
  13. Micro-Benchmarks
  14. Benefit of EENs
  15. Necessity of Cross-Client KD
  16. ...and 17 more sections

Key Result

Proposition 1

Given an input sample $x$, the cross-client KD in Eq. (equ:cc-kd) is equivalent to knowledge distillation from an aggregated teacher model $\bar{\theta} = (\phi, \bar{h})$ where $\bar{h} = \sum_{i=1}^n k_i h_{im}$.

Figures (10)

  • Figure 1: Motivation for personalized federated learning of early-exit networks (PFL-EE).
  • Figure 2: An empirical comparison of post-PFL exit training (local), direct extension of PFL to EENs (joint), existing GFL-EE (joint + local KD), and our solution (joint + cross-client KD) to train personalized 4-exit EENs for 100 clients on CIFAR-100 under Dir(0.3).
  • Figure 3: Overview of CAFEDistill.
  • Figure 4: Conflict-aware student selection.
  • Figure 5: Decomposition of cross-client KD.
  • ...and 5 more figures

Theorems & Definitions (2)

  • Proposition 1: Equivalence of KD
  • Proposition 2: Monotonicity and submodularity of $f(\mathcal{E})$