Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generating customized prompts for Zero-Shot Rare Event Medical Image Classification using LLM

Payal Kamboj, Ayan Banerjee, Bin Xu, Sandeep Gupta

TL;DR

The paper tackles the problem of rare medical event classification where data is scarce, making traditional deep learning unreliable. It introduces CuKPL, a framework that generates Human Knowledge-Embodied Textual Prompts (HKETP) by refining domain knowledge with an LLM and encoding it into discriminative prompts that can be used by LLMs for zero-shot classification without accessing the images. HKETP, contextualized with instruction and human verifier feedback, enables privacy-preserving, image-free inference and leverages domain knowledge to outperform state-of-the-art methods on SOZ detection. The results demonstrate strong within-domain performance and notable cross-center generalization, with potential applicability to other medical imaging tasks.

Abstract

Rare events, due to their infrequent occurrences, do not have much data, and hence deep learning techniques fail in estimating the distribution for such data. Open-vocabulary models represent an innovative approach to image classification. Unlike traditional models, these models classify images into any set of categories specified with natural language prompts during inference. These prompts usually comprise manually crafted templates (e.g., 'a photo of a {}') that are filled in with the names of each category. This paper introduces a simple yet effective method for generating highly accurate and contextually descriptive prompts containing discriminative characteristics. Rare event detection, especially in medicine, is more challenging due to low inter-class and high intra-class variability. To address these, we propose a novel approach that uses domain-specific expert knowledge on rare events to generate customized and contextually relevant prompts, which are then used by large language models for image classification. Our zero-shot, privacy-preserving method enhances rare event classification without additional training, outperforming state-of-the-art techniques.

Generating customized prompts for Zero-Shot Rare Event Medical Image Classification using LLM

TL;DR

The paper tackles the problem of rare medical event classification where data is scarce, making traditional deep learning unreliable. It introduces CuKPL, a framework that generates Human Knowledge-Embodied Textual Prompts (HKETP) by refining domain knowledge with an LLM and encoding it into discriminative prompts that can be used by LLMs for zero-shot classification without accessing the images. HKETP, contextualized with instruction and human verifier feedback, enables privacy-preserving, image-free inference and leverages domain knowledge to outperform state-of-the-art methods on SOZ detection. The results demonstrate strong within-domain performance and notable cross-center generalization, with potential applicability to other medical imaging tasks.

Abstract

Rare events, due to their infrequent occurrences, do not have much data, and hence deep learning techniques fail in estimating the distribution for such data. Open-vocabulary models represent an innovative approach to image classification. Unlike traditional models, these models classify images into any set of categories specified with natural language prompts during inference. These prompts usually comprise manually crafted templates (e.g., 'a photo of a {}') that are filled in with the names of each category. This paper introduces a simple yet effective method for generating highly accurate and contextually descriptive prompts containing discriminative characteristics. Rare event detection, especially in medicine, is more challenging due to low inter-class and high intra-class variability. To address these, we propose a novel approach that uses domain-specific expert knowledge on rare events to generate customized and contextually relevant prompts, which are then used by large language models for image classification. Our zero-shot, privacy-preserving method enhances rare event classification without additional training, outperforming state-of-the-art techniques.

Paper Structure

This paper contains 6 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Application Domain: Rare SOZ Detection
  5. Experimental Setup and Evaluation
  6. Conclusion

Figures (2)

  • Figure 1: Example CuPL LLM-prompts, Image prompts (Top), and CuKPL LLM prompts and classification (Bottom).
  • Figure 2: a) HKETP Generation: Raw knowledge refined by LLM is encoded using image processing techniques in knowledge model, producing HKETP. Refined knowledge (Context), HKETP (Input) along with Instruction are used by LLM for classification, with feedback from a human verifier. b) Inference: Given an image, Knowledge model outputs input HKETP which combined with refined knowledge context and instruction is collectively used by LLM to classify rare event.