Query-based Knowledge Transfer for Heterogeneous Learning Environments

TL;DR

The paper addresses customized query-driven knowledge transfer in decentralized learning under data heterogeneity and privacy constraints. It introduces Query-based Knowledge Transfer (QKT), a data-free, two-phase framework that uses synthetic data-driven masks for query-focused distillation (Phase 1) and subsequent Classification Head Refinement (Phase 2) to mitigate forgetting. Empirical results on standard and clinical benchmarks show substantial improvements over federated, KD, and ensemble baselines, with average gains around 20.9 points for single-class and 14.3 points for multi-class queries, while reducing communication overhead. A lighter variant, QKT Light, offers further efficiency, and ablations highlight the critical roles of masking, head refinement, and two-phase stability for robust decentralized learning.

Abstract

Decentralized collaborative learning under data heterogeneity and privacy constraints has rapidly advanced. However, existing solutions like federated learning, ensembles, and transfer learning, often fail to adequately serve the unique needs of clients, especially when local data representation is limited. To address this issue, we propose a novel framework called Query-based Knowledge Transfer (QKT) that enables tailored knowledge acquisition to fulfill specific client needs without direct data exchange. QKT employs a data-free masking strategy to facilitate communication-efficient query-focused knowledge transfer while refining task-specific parameters to mitigate knowledge interference and forgetting. Our experiments, conducted on both standard and clinical benchmarks, show that QKT significantly outperforms existing collaborative learning methods by an average of 20.91\% points in single-class query settings and an average of 14.32\% points in multi-class query scenarios. Further analysis and ablation studies reveal that QKT effectively balances the learning of new and existing knowledge, showing strong potential for its application in decentralized learning.

Figures (7)

• Figure 1: Extraneous knowledge, whether from additional classes or non-proficient teachers, can interfere with the query and local classes, leading to unsatisfactory performance for the query classes.
• Figure 2: Decision boundaries and class accuracy of Local Training, Naive KD, QKT Phase 1, and QKT Phase 2. Refining the classification head in QKT Phase 2 markedly improves the performance.
• Figure 3: Query-based Knowledge Transfer (QKT): noise is applied to estimate the relevance of the teacher models to the student's query to obtain masks for each teacher model. Masked distillation is then applied to transfer the knowledge of the query classes from the teacher models to the student model. We then refine the classification head of the student model to prevent forgetting the previously learned knowledge.
• Figure 4: Forgetting vs. Query Accuracy Gain across datasets. Each column shows one dataset, while rows correspond to different data distributions (Pathological/Dirichlet) and query types (Single-Class/Multi-Class). Points closer to the upper-right corner represent lower forgetting and higher query accuracy gain.
• Figure 5: Comparison of actual data distribution (left) and predicted probabilities (right) for each client using noise-based filtering. The left heatmap is normalized to facilitate comparison, and the prediction error is shown in MSE.