Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring connections of spectral analysis and transfer learning in medical imaging

Yucheng Lu, Dovile Juodelyte, Jonathan D. Victor, Veronika Cheplygina

TL;DR

This work examines how transfer learning for medical imaging is influenced by spectral properties of pre-trained models, particularly their learning priorities across frequency bands. By analyzing gradient-based power spectrum density ($\mathrm{PSD}$) and introducing artificial frequency shortcuts, the authors show clear differences between models pre-trained on natural data (ImageNet) versus medical data (RadImageNet), and that these priors persist during fine-tuning. They demonstrate that alignment between a model’s learning priorities and the spectral content of artifacts leads to shortcut learning, and that source data editing can modulate robustness to these shortcuts. The findings suggest spectral-domain interventions in pre-training could improve transfer-learning robustness in medical imaging, while also raising questions about how source data statistics shape learning priors and how to manipulate them safely and effectively.

Abstract

In this paper, we use spectral analysis to investigate transfer learning and study model sensitivity to frequency shortcuts in medical imaging. By analyzing the power spectrum density of both pre-trained and fine-tuned model gradients, as well as artificially generated frequency shortcuts, we observe notable differences in learning priorities between models pre-trained on natural vs medical images, which generally persist during fine-tuning. We find that when a model's learning priority aligns with the power spectrum density of an artifact, it results in overfitting to that artifact. Based on these observations, we show that source data editing can alter the model's resistance to shortcut learning.

Exploring connections of spectral analysis and transfer learning in medical imaging

TL;DR

This work examines how transfer learning for medical imaging is influenced by spectral properties of pre-trained models, particularly their learning priorities across frequency bands. By analyzing gradient-based power spectrum density () and introducing artificial frequency shortcuts, the authors show clear differences between models pre-trained on natural data (ImageNet) versus medical data (RadImageNet), and that these priors persist during fine-tuning. They demonstrate that alignment between a model’s learning priorities and the spectral content of artifacts leads to shortcut learning, and that source data editing can modulate robustness to these shortcuts. The findings suggest spectral-domain interventions in pre-training could improve transfer-learning robustness in medical imaging, while also raising questions about how source data statistics shape learning priors and how to manipulate them safely and effectively.

Abstract

In this paper, we use spectral analysis to investigate transfer learning and study model sensitivity to frequency shortcuts in medical imaging. By analyzing the power spectrum density of both pre-trained and fine-tuned model gradients, as well as artificially generated frequency shortcuts, we observe notable differences in learning priorities between models pre-trained on natural vs medical images, which generally persist during fine-tuning. We find that when a model's learning priority aligns with the power spectrum density of an artifact, it results in overfitting to that artifact. Based on these observations, we show that source data editing can alter the model's resistance to shortcut learning.
Paper Structure (15 sections, 2 equations, 4 figures)

This paper contains 15 sections, 2 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Frequency bias
  4. Frequency shortcut
  5. Spectrum augmentation
  6. Methodology
  7. Datasets and models
  8. Frequency shortcuts
  9. Power spectrum density
  10. Experimental results
  11. ImageNet is prone to shortcut learning
  12. Learning priority is stable during transfer
  13. PSD is related to shortcut learning
  14. Source data affects robustness
  15. Conclusions and Future Work

Figures (4)

  • Figure 1: Example of a PSD. From left to right: original image, its spectrum with a selected frequency $\omega_{128}$, and the PSD with the highlighted frequency $\omega_{128}$.
  • Figure 2: Baseline results (mean and standard deviation of AUC across 5-folds) as a function of degradation (amount of artifacts in the training set), performance on O.O.D. (top) and I.I.D. (bottom) test sets.
  • Figure 3: Normalized learning priorities of pre-trained and fine-tuned models. $\bar{\omega}_{k}$ is the normalized radical frequency with respect to the highest value (x-axis shared between rows). Top: normalized PSDs from Eq. 1. The arrows show how the pre-trained model PSDs change before and after source data editing. Middle: PSD as a heat map, after different degrees of degradation (amount of artifacts in the training set) for the original datasets. Bottom: Same as above but for the edited datasets.
  • Figure 4: O.O.D. performance (mean and standard deviation of AUC across 5-folds) with (doted lines) and without (solid lines) source data editing.