PM2: A New Prompting Multi-modal Model Paradigm for Few-shot Medical Image Classification

TL;DR

PM2 introduces a prompting multi-modal paradigm for few-shot medical image classification, augmenting image data with text prompts and a covariance-based visual head to exploit second-order feature statistics. Leveraging a frozen CLIP backbone, PM2 uses a shared classifier to fuse image-derived class tokens and text-prompt representations, while a covariance pooling step with Newton-Schulz normalization captures rich visual token relationships. Five text-prompt schemes are explored (including CoOp and GPT-based prompts) with extensive ablations, and the approach achieves state-of-the-art results on BACH, brain MRI, and DR datasets under few-shot settings. This work demonstrates that multi-modal prompting and second-order feature modeling can substantially reduce annotation requirements while delivering robust medical image classification performance, with broad implications for data-efficient, cross-modal diagnostic tools.

Abstract

Few-shot learning has been successfully applied to medical image classification as only very few medical examples are available for training. Due to the challenging problem of limited number of annotated medical images, image representations should not be solely derived from a single image modality which is insufficient for characterizing concept classes. In this paper, we propose a new prompting multi-modal model paradigm on medical image classification based on multi-modal foundation models, called PM2. Besides image modality,PM2 introduces another supplementary text input, known as prompt, to further describe corresponding image or concept classes and facilitate few-shot learning across diverse modalities. To better explore the potential of prompt engineering, we empirically investigate five distinct prompt schemes under the new paradigm. Furthermore, linear probing in multi-modal models acts as a linear classification head taking as input only class token, which ignores completely merits of rich statistics inherent in high-level visual tokens. Thus, we alternatively perform a linear classification on feature distribution of visual tokens and class token simultaneously. To effectively mine such rich statistics, a global covariance pooling with efficient matrix power normalization is used to aggregate visual tokens. Then we study and combine two classification heads. One is shared for class token of image from vision encoder and prompt representation encoded by text encoder. The other is to classification on feature distribution of visual tokens from vision encoder. Extensive experiments on three medical datasets show that our PM2 significantly outperforms counterparts regardless of prompt schemes and achieves state-of-the-art performance.

Figures (2)

• Figure 1: Schematic diagram of the PM$^2$ structure. First, a prompting method is selected from the Prompt Pool to generate prompts for each class, using the CoOp method as example in the diagram. Then, visual and textual features are extracted using their respective encoders, with ViT as example of a visual encoder. Subsequently, a shared classifier predicts classes based on text features ([EOS] token) and a visual class token. For the visual part's class prediction, covariance pooling of visual tokens is further calculated for second-order statistical modeling. The second-order prediction is then added to the first-order prediction, represented by a single class token, to form the final prediction for the visual component. The text encoder and image encoder are frozen during training for efficiency.
• Figure 2: Schematic diagram of the visual classification head. The visual classification head aggregates the visual tokens output by the image encoder, differing from the naive use of global average pooling (GAP), as shown in the upper part of the diagram. This paper employs covariance pooling for second-order statistical modeling of visual tokens and combines this with the first-order information represented by a single class token for the visual component's prediction, as shown in the lower part of the diagram.