Beyond Knowledge Silos: Task Fingerprinting for Democratization of Medical Imaging AI

TL;DR

This work tackles knowledge silos in medical imaging AI by introducing task fingerprinting, a privacy-preserving framework that builds shareable task representations to guide cross-task knowledge transfer via a knowledge cloud. Central to the method is the binned Kullback-Leibler Divergence (bKLD) distance, which weights per-feature histograms to robustly compare tasks across diverse modalities and architectures. Across 71 tasks in 12 modalities and four transfer scenarios (architecture, pretraining data, augmentation policies, and co-training data), fingerprinting outperforms manual task selection and existing task similarity measures, with strong robustness to fingerprint size and demonstrated generalization to unseen task types and 3D segmentation. The approach promises democratization of medical imaging AI by enabling collaborative, data-efficient model development and potentially reducing training overhead and energy use. Future directions include deploying a knowledge cloud, exploring entangled transfers, and expanding to broader modalities and foundation-model integration.

Abstract

The field of medical imaging AI is currently undergoing rapid transformations, with methodical research increasingly translated into clinical practice. Despite these successes, research suffers from knowledge silos, hindering collaboration and progress: Existing knowledge is scattered across publications and many details remain unpublished, while privacy regulations restrict data sharing. In the spirit of democratizing of AI, we propose a framework for secure knowledge transfer in the field of medical image analysis. The key to our approach is dataset "fingerprints", structured representations of feature distributions, that enable quantification of task similarity. We tested our approach across 71 distinct tasks and 12 medical imaging modalities by transferring neural architectures, pretraining, augmentation policies, and multi-task learning. According to comprehensive analyses, our method outperforms traditional methods for identifying relevant knowledge and facilitates collaborative model training. Our framework fosters the democratization of AI in medical imaging and could become a valuable tool for promoting faster scientific advancement.

Figures (8)

• Figure 1: Concept of task fingerprinting for knowledge transfer. (I) Participants of the knowledge cloud contribute by submitting a shareable task representation ("fingerprint"), meta information about their training strategies, and, optionally, their data. (II) A user generates the fingerprint for their task and queries the knowledge cloud. (III) Based on the most relevant tasks in the pool according to fingerprint matching, relevant training strategies and data can be retrieved. (IV) The retrieved meta information and data are used to compile a training pipeline with different components of transferred knowledge. In this study, we investigate four scenarios of knowledge transfer, namely (a) model architecture, (b) pretraining data, (c) augmentation policy, and (d) co-training data.
• Figure 2: Proposed fingerprinting strategy and associated task distance measure: binned Kullback-Leibler Divergence (bKLD).Top: To compute a task fingerprint, $n=10,000$ images are sampled (I) and passed through a pretrained backbone (we use an ImageNet Deng2009ImageNetAL ResNet34 He2016DeepRL) to extract $m=512$ features per image (II). The resulting $m \times n$ features are binned into $b=100$ bins along the features axis (III), resulting in m (normalized) histograms representing the fingerprint. Bottom: To compare two fingerprints, a weighted sum of the Kullback-Leibler Divergence (KLD) across all per-feature histograms is computed. A softmax operation is applied to source task histograms to avoid empty bins.
• Figure 3: Our study is based on a heterogeneous set of 71 imaging tasks. For each of the 28 tasks of the development split as well as 43 tasks of the validation split, we show a sample image next to the distribution of the following Box-Cox and Z-score transformed properties: number of classes, number of samples, intrinsic data dimension Pope2021dimension, and imbalance ratio (size of largest class divided by size of the smallest class). The imaging domain is encoded as the background color of the dataset name.
• Figure 4: Task fingerprinting benefits training pipeline configuration and beats manual knowledge transfer.(a) Top: Fraction of n=43 validation tasks that improve Balanced Accuracy (BA) through knowledge transfer ("gain" Zamir2018TaskonomyDT) in four scenarios. (b) Bottom: Average delta in Area Under the Receiver Operator Characteristic (AUROC) across n=43 validation tasks. X-axis shows the number of shots, translating to the best of top k suggestions of our framework. Error bars correspond to standard deviation over three repetitions of all model trainings. Our proposed binned Kullback-Leibler Divergence (bKLD) fingerprint (here: the small variant) improves training for up to 90% of validation tasks.
• Figure 5: binned Kullback-Leibler Divergence (bKLD) outperforms previously proposed methods for knowledge transfer. Uncertainty-aware ranking of our proposed bKLD methods for task fingerprinting versus manual task selection, VisualDNA (VDNA) Ramtoula2023VisualDR, Fisher Embedding Distance (FED) Godau2021TaskFingachille2019task2vec, Fréchet Inception Distance (FID) Heusel2017FIDDing2021AnalyzingDN, Predict To Learn (P2L) Bhattacharjee2020P2LPT, and Sinkhorn Divergence feydy2019interpolating. (a)Bottom: Columns represent four meta metrics to compare distance measures, whilst rows correspond to four knowledge transfer scenarios. We average across the top three suggestions by each method (weightedtau evaluates the full candidate ranking). Blob size shows frequency of rank across 1000 bootstrap samples from 258 setups (2 base metrics, 43 validation tasks, 3 repetitions). X marks the mean rank and whiskers the standard deviation. Plot is inspired by Wiesenfarth2021RR and follows the “aggregate then rank" assessment method. (b)Top: Summary of the 16 subplots below. The marker position refers to the mean rank over individual bootstrapped mean rankings (X marks) and whiskers indicate standard deviation.