Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for zero-shot medical detection

TL;DR

StructuralGLIP targets the problem of coarse alignment between visual features and target disease descriptions in zero-shot medical detection by introducing a dual-branch architecture with a latent knowledge bank that stores prompts. Through layer-wise mutual selection of structurally meaningful visual and linguistic tokens, it enables context-aware cross-modal fusion and supports both instance- and category-level prompts, further enhanced by instance-specific VQA prompts and category-level prompts from large language models. Empirical results across eight medical benchmarks show consistent zero-shot gains (about +4% AP on average) and substantial improvements when used to enhance fine-tuned GLIP models, with category-level prompts achieving particularly strong performance while maintaining efficiency. The approach is robust to noisy prompts and provides insights into hierarchical prompt utilization, with practical implications for expanding zero-shot medical detection in clinical settings and beyond.

Abstract

Zero-shot medical detection can further improve detection performance without relying on annotated medical images even upon the fine-tuned model, showing great clinical value. Recent studies leverage grounded vision-language models (GLIP) to achieve this by using detailed disease descriptions as prompts for the target disease name during the inference phase. However, these methods typically treat prompts as equivalent context to the target name, making it difficult to assign specific disease knowledge based on visual information, leading to a coarse alignment between images and target descriptions. In this paper, we propose StructuralGLIP, which introduces an auxiliary branch to encode prompts into a latent knowledge bank layer-by-layer, enabling more context-aware and fine-grained alignment. Specifically, in each layer, we select highly similar features from both the image representation and the knowledge bank, forming structural representations that capture nuanced relationships between image patches and target descriptions. These features are then fused across modalities to further enhance detection performance. Extensive experiments demonstrate that StructuralGLIP achieves a +4.1\% AP improvement over prior state-of-the-art methods across seven zero-shot medical detection benchmarks, and consistently improves fine-tuned models by +3.2\% AP on endoscopy image datasets.

Figures (10)

• Figure 1: (a) Contextual prompt methods directly concatenate the prompt and target. (b) Our structural prompt method encodes prompts into a latent knowledge bank.
• Figure 2: Experimental settings for zero-shot medical detection and enhancement.
• Figure 3: Pipeline of the proposed method and the automatic prompt generation.
• Figure 4: Ablation towards the fusing layer of the proposed method
• Figure A: Visualization Results on the ColonDB and BCCD datasets.