G2L:From Giga-Scale to Cancer-Specific Large-Scale Pathology Foundation Models via Knowledge Distillation

TL;DR

The paper tackles the impractical resource demands of giga-scale pathology foundation models by introducing G2L, a knowledge-distillation framework that transfers knowledge from a giga-scale teacher to a cancer-specific large-scale student using only 1K slides. The approach uses patch-based sampling, a linear projection to align embeddings, and a Log-Sum KD loss, achieving giga-scale-level performance on multiple cancer benchmarks with only 15% of the teacher's parameters. Across extensive experiments, the G2L-distilled model not only matches or surpasses large-scale baselines but also exceeds certain giga- and huge-scale models on several tasks, while demonstrating superior robustness to inter-center variability. These results suggest a data- and compute-efficient path to clinically actionable, cancer-specific pathology foundation models.

Abstract

Recent studies in pathology foundation models have shown that scaling training data, diversifying cancer types, and increasing model size consistently improve their performance. However, giga-scale foundation models, which are trained on hundreds of thousands of slides covering tens of cancer types and contain billions of parameters, pose significant challenges for practical use due to their tremendous computational costs in both development and deployment. In this work, we present a novel strategy, named the G2L framework, to increase the performance of large-scale foundation models, which consist of only $15\%$ of the parameters of giga-scale models, to a comparable performance level of giga-scale models in cancer-specific tasks. Our approach applies knowledge distillation, transferring the capabilities of a giga-scale model to a large-scale model, using just 1K pathology slides of a target cancer (e.g., breast, prostate, etc.). The resulting distilled model not only outperformed state-of-the-art models of the same size (i.e., large-scale) across several benchmarks but also, interestingly, surpassed the giga-scale teacher and huge-scale models in some benchmarks. In addition, the distilled model exhibited a higher robustness index, indicating improved resilience to image variations originating from multiple institutions. These findings suggest that the proposed distillation approach for a large-scale model is a data- and parameter-efficient way to achieve giga-scale-level performance for cancer-specific applications without prohibitive computational burden.

Figures (2)

• Figure 1: Comparison of G2L-distilled and other foundation models in TP53 mutation prediction benchmark measured by accuracy with model parameters (in billion) and inference times (in seconds). The G2L-optimized model outperforms the others in all metrics, as highlighted in red.
• Figure 2: Overview of G2L framework.