Memory-Efficient Prompt Tuning for Incremental Histopathology Classification

TL;DR

This paper tackles histopathology classification under domain-incremental learning with privacy constraints by introducing a memory-efficient prompt-tuning framework. It freezes the backbone and attaches two prompts per domain: a domain-specific prompt (DSP) and a domain-invariant prompt (DIP) that evolves over time, with the DIP refined via style-augmented graph attention to capture domain-generic representations and the DSP stored to prevent forgetting. A prompt bank stores all prompts; the DIP evolves through a graph attention mechanism using style-augmented data to improve generalization to unseen domains, while DSPs isolate domain-specific features. Empirical results on Camelyon17 breast cancer metastasis and epithelium-stroma tissue classification show superior accuracy, reduced forgetting, and favorable memory efficiency compared to state-of-the-art incremental learning methods, highlighting practical benefits for privacy-sensitive medical analytics.

Abstract

Recent studies have made remarkable progress in histopathology classification. Based on current successes, contemporary works proposed to further upgrade the model towards a more generalizable and robust direction through incrementally learning from the sequentially delivered domains. Unlike previous parameter isolation based approaches that usually demand massive computation resources during model updating, we present a memory-efficient prompt tuning framework to cultivate model generalization potential in economical memory cost. For each incoming domain, we reuse the existing parameters of the initial classification model and attach lightweight trainable prompts into it for customized tuning. Considering the domain heterogeneity, we perform decoupled prompt tuning, where we adopt a domain-specific prompt for each domain to independently investigate its distinctive characteristics, and one domain-invariant prompt shared across all domains to continually explore the common content embedding throughout time. All domain-specific prompts will be appended to the prompt bank and isolated from further changes to prevent forgetting the distinctive features of early-seen domains. While the domain-invariant prompt will be passed on and iteratively evolve by style-augmented prompt refining to improve model generalization capability over time. In specific, we construct a graph with existing prompts and build a style-augmented graph attention network to guide the domain-invariant prompt exploring the overlapped latent embedding among all delivered domains for more domain generic representations. We have extensively evaluated our framework with two histopathology tasks, i.e., breast cancer metastasis classification and epithelium-stroma tissue classification, where our approach yielded superior performance and memory efficiency over the competing methods.

Figures (4)

• Figure 1: Overview of our memory-efficient prompt tuning framework. We proposed to perform decoupled prompt tuning upon the initial model with two lightweight prompts, aiming to acquire the latest domain knowledge in economical memory cost. We employ a domain-specific prompt independently for each domain to acquire its distinctive features like appearances. The learned domain-specific prompt would be stored and isolated in the prompt bank to help alleviate the forgetting of early-acquired domains. Meanwhile, we maintain a domain-invariant prompt shared across domains to progressively learn the common content over time like shape prior. We performed style-augmented prompt refining upon the previous domain-invariant prompt, where we constrain its exploration scope within the overlapped latent embeddings of all seen domains and guide it to learn the domain-generic representations, gradually strengthening the generalization potential over time.
• Figure 2: Illustration of generating style-augmented data.
• Figure 3: The t-SNE visualization of the feature embeddings after applying decoupled prompt tuning.
• Figure 4: Searching results for the inserted layer of prompts