Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Soft-Label Caching and Sharpening for Communication-Efficient Federated Distillation

Kitsuya Azuma, Takayuki Nishio, Yuichi Kitagawa, Wakako Nakano, Takahito Tanimura

TL;DR

This paper tackles the high communication cost and heterogeneity challenges in federated distillation by introducing SCARLET, a framework that caches soft-labels across rounds and uses Enhanced ERA for robust, entropy-conscious aggregation. The approach reduces redundant data transmission while maintaining competitive accuracy, outperforming state-of-the-art distillation-based FL methods in non-IID settings. Extensive experiments across CIFAR-10/100 and Tiny ImageNet demonstrate significant uplink savings and stable convergence, with ablations clarifying the roles of cache duration and the beta-controlled Enhanced ERA. The work also provides practical insights for partial participation and offers an open-source implementation to facilitate reproducibility and further research.

Abstract

Federated Learning (FL) enables collaborative model training across decentralized clients, enhancing privacy by keeping data local. Yet conventional FL, relying on frequent parameter-sharing, suffers from high communication overhead and limited model heterogeneity. Distillation-based FL approaches address these issues by sharing predictions (soft-labels, i.e., normalized probability distributions) instead, but they often involve redundant transmissions across communication rounds, reducing efficiency. We propose SCARLET, a novel framework integrating synchronized soft-label caching and an enhanced Entropy Reduction Aggregation (Enhanced ERA) mechanism. SCARLET minimizes redundant communication by reusing cached soft-labels, achieving up to 50% reduction in communication costs compared to existing methods while maintaining competitive accuracy. Enhanced ERA resolves the fundamental instability of conventional temperature-based aggregation, ensuring robust control and high performance in diverse client scenarios. Experimental evaluations demonstrate that SCARLET consistently outperforms state-of-the-art distillation-based FL methods in terms of accuracy and communication efficiency. The implementation of SCARLET is publicly available at https://github.com/kitsuyaazuma/SCARLET.

Soft-Label Caching and Sharpening for Communication-Efficient Federated Distillation

TL;DR

This paper tackles the high communication cost and heterogeneity challenges in federated distillation by introducing SCARLET, a framework that caches soft-labels across rounds and uses Enhanced ERA for robust, entropy-conscious aggregation. The approach reduces redundant data transmission while maintaining competitive accuracy, outperforming state-of-the-art distillation-based FL methods in non-IID settings. Extensive experiments across CIFAR-10/100 and Tiny ImageNet demonstrate significant uplink savings and stable convergence, with ablations clarifying the roles of cache duration and the beta-controlled Enhanced ERA. The work also provides practical insights for partial participation and offers an open-source implementation to facilitate reproducibility and further research.

Abstract

Federated Learning (FL) enables collaborative model training across decentralized clients, enhancing privacy by keeping data local. Yet conventional FL, relying on frequent parameter-sharing, suffers from high communication overhead and limited model heterogeneity. Distillation-based FL approaches address these issues by sharing predictions (soft-labels, i.e., normalized probability distributions) instead, but they often involve redundant transmissions across communication rounds, reducing efficiency. We propose SCARLET, a novel framework integrating synchronized soft-label caching and an enhanced Entropy Reduction Aggregation (Enhanced ERA) mechanism. SCARLET minimizes redundant communication by reusing cached soft-labels, achieving up to 50% reduction in communication costs compared to existing methods while maintaining competitive accuracy. Enhanced ERA resolves the fundamental instability of conventional temperature-based aggregation, ensuring robust control and high performance in diverse client scenarios. Experimental evaluations demonstrate that SCARLET consistently outperforms state-of-the-art distillation-based FL methods in terms of accuracy and communication efficiency. The implementation of SCARLET is publicly available at https://github.com/kitsuyaazuma/SCARLET.
Paper Structure (45 sections, 1 theorem, 24 equations, 18 figures, 6 tables, 3 algorithms)

This paper contains 45 sections, 1 theorem, 24 equations, 18 figures, 6 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Federated Learning (FL)
  4. Communication-Efficient FL
  5. Distillation-Based FL
  6. FL with Caching Mechanisms
  7. Proposed Method
  8. Background and Notations
  9. SCARLET: System Design and Workflow
  10. Soft-Label Caching in SCARLET
  11. Handling Partial Client Participation
  12. Enhanced ERA in SCARLET
  13. Experiments
  14. Experimental Setup
  15. Datasets and Partitioning
  16. ...and 30 more sections

Key Result

Theorem 1

Let $\{x_i\}_{i=1}^N$ be nonnegative real numbers in non-decreasing order, and let For any $\beta_2 > \beta_1 > 0$, define the unnormalized vectors and their corresponding normalized distributions Then, for each $1 \le k \le N$, the following holds: Equivalently, when viewed as probability distributions, $\hat{X}^{(\beta_2)}$ is majorized by $\hat{X}^{(\beta_1)}$.

Figures (18)

  • Figure 1: Conceptual overview of distillation-based FL approaches. Conventional distillation-based FL (\ref{['fig:concept:distillation_based_fl']}) transmits all soft-labels in each communication round. SCARLET (\ref{['fig:concept:scarlet']}) requests soft-labels only for data not present in the server cache and synchronizes cache updates between the server and clients, significantly reducing communication costs.
  • Figure 2: Illustration of the soft-label caching mechanism in SCARLET. When public data is not cached (\ref{['fig:detail:not_cached']}), clients transmit their soft-labels to the server, where they are aggregated using Enhanced ERA and stored in the server cache before synchronizing with client caches. When public data is cached (\ref{['fig:detail:cached']}), both the server and clients retrieve soft-labels from their respective caches for knowledge distillation, eliminating the need for additional soft-label communication.
  • Figure 3: Simulation of the ratio of cached samples per round as a function of cache duration $D$. Cached samples are defined as those in $\mathcal{P}^t$ that are found in the local cache and have not expired. The communication cost decreases as the ratio of cached samples increases, since only uncached samples require transmission.
  • Figure 4: Comparison of ERA and Enhanced ERA with different parameter settings. (\ref{['fig:comparison_era_enhanced_era:high-entropy']}) and (\ref{['fig:comparison_era_enhanced_era:low-entropy']}) show the original high-entropy (low-confidence) and low-entropy (high-confidence) soft-labels, respectively. (\ref{['fig:comparison_era_enhanced_era:era']}) illustrates how ERA rapidly reduces entropy by decreasing temperature $T$, particularly for (\ref{['fig:comparison_era_enhanced_era:high-entropy']}) compared to (\ref{['fig:comparison_era_enhanced_era:low-entropy']}). (\ref{['fig:comparison_era_enhanced_era:enhanced_era']}) demonstrates how Enhanced ERA provides smoother and more flexible entropy control by varying $\beta$, ensuring that $\beta = 1$ recovers the original entropy for both (\ref{['fig:comparison_era_enhanced_era:high-entropy']}) and (\ref{['fig:comparison_era_enhanced_era:low-entropy']}).
  • Figure 5: Preliminary comparison of aggregation stability. Learning curves show Enhanced ERA (red lines) consistently outperforms the conventional ERA (gray lines) on CIFAR-10. The optimal parameters plotted are Enhanced ERA ($\beta=2.5$) vs. ERA ($T=0.1$) for strong non-IID (\ref{['fig:era_vs_enhanced_era:0-05']}), and Enhanced ERA ($\beta=1.0$) vs. ERA ($T=0.2$) for moderate non-IID (\ref{['fig:era_vs_enhanced_era:0-3']}).
  • ...and 13 more figures

Theorems & Definitions (2)

  • Theorem 1: Majorization via Power Exponents
  • proof