InvCoSS: Inversion-driven Continual Self-supervised Learning in Medical Multi-modal Image Pre-training

TL;DR

InvCoSS tackles catastrophic forgetting in medical multi-modal pre-training without accessing past raw data by synthesizing past-task images through inversion of a frozen model. It introduces InvUNet to recover high-frequency details, and repulsive representation learning to maintain diversity, collectively enabling data-free continual SSL that rivals data-replay methods in nine downstream tasks. The approach dramatically reduces storage needs (up to 590×) and preserves privacy by avoiding real past data, while maintaining strong performance across modalities. This work demonstrates a practical, privacy-aware pathway for scalable continual pre-training in healthcare imaging.

Abstract

Continual self-supervised learning (CSSL) in medical imaging trains a foundation model sequentially, alleviating the need for collecting multi-modal images for joint training and offering promising improvements in downstream performance while preserving data privacy. However, most existing methods still rely on replaying data from previous stages to prevent catastrophic forgetting, which compromises privacy and limits their applicability in real-world scenarios where data transfer across sites is often restricted. In this work, we propose InvCoSS, an inversion-driven continual self-supervised learning framework for medical multi-modal image pre-training. Specifically, after training on a previous task, InvCoSS inverts the pre-trained self-supervised model to generate synthetic images that approximate the original training distribution. These synthetic images are then combined with data from the new task for joint optimization, which effectively mitigates catastrophic forgetting while strictly adhering to the constraint of no access to previous real data. Furthermore, to improve the fidelity of synthetic images, we introduce a novel InvUNet with a multi-scale fusion architecture to restore both high- and low-frequency components of the inverted images. To enhance diversity and prevent mode collapse, we design a repulsive representation-learning mechanism that encourages a diverse feature space for synthetic images without class guidance. Extensive experiments across nine downstream tasks validate the effectiveness of InvCoSS, achieving performance comparable to or even superior to prior data-replay methods while significantly reducing storage requirements and eliminating data privacy constraints.

Figures (10)

• Figure 1: We propose InvCoSS, a novel inversion-driven continual self-supervised learning (CSSL) framework that utilizes synthetic images inverted from just self-supervised models as continual knowledge retention. Besides, we design InvUNet (Sec.\ref{['sec:invunet']}) with multi-scale fusion to preserve high-frequency details, and introduce repulsive representation learning (Sec.\ref{['sec:rrl']}) to explicitly enforce diversity and prevent mode collapse, alongside feature regularization, task objectives, and image prior(Sec.\ref{['sec:ooo']}). Using these synthetic images for CSSL, we effectively mitigate catastrophic forgetting while eliminating data privacy constraints across institutes.
• Figure 2: Overview of inversion-based image synthesis. InvUNet synthesizes images from a bottleneck injected latent $z$, with a memory-cache branch providing multi-scale priors via skip connections. A frozen model $f_{T-1}$ supervises with MIM task prior $\mathcal{L}_{\mathrm{task}}$, norm statistics matching $\mathcal{L}_{\mathrm{norm}}$, and total variation image prior $\mathcal{L}_{\mathrm{img}}$, while a persistent feature pool imposes a repulsive loss $\mathcal{L}_{\mathrm{rep}}$ to promote diversity. The process is data-free and produces synthetic datasets $D_t^{syn}$ for knowledge retention at stage T.
• Figure 3: Visualizations of synthetic data generated by our inversion-driven framework, showcasing its versatility across diverse medical tasks. The figure displays samples from four imaging modalities (two columns each): X-ray, CT, MR, and Pathological imaging.
• Figure 4: t-SNE visualization of real and synthetic image feature distributions. Left: X-ray. Right: Path. Gray: original data. Blue/Red: synthetic images with/without $\mathcal{L}_{rep}$.
• Figure 5: Visual comparison of synthetic images generated with different configurations. From top to bottom: without generator, without image prior, with bottom-to-top generator, and our InvUNet. Each row shows X-ray and pathological imaging samples.