AttriPrompter: Auto-Prompting with Attribute Semantics for Zero-shot Nuclei Detection via Visual-Language Pre-trained Models

TL;DR

An innovative auto-prompting pipeline, named AttriPrompter, comprising attribute generation, attribute augmentation, and relevance sorting, to avoid subjective manual prompt design is proposed, to investigate the potential of the object-level VLPM, Grounded Language-Image Pre-training (GLIP), for zero-shot nuclei detection.

Abstract

Large-scale visual-language pre-trained models (VLPMs) have demonstrated exceptional performance in downstream object detection through text prompts for natural scenes. However, their application to zero-shot nuclei detection on histopathology images remains relatively unexplored, mainly due to the significant gap between the characteristics of medical images and the web-originated text-image pairs used for pre-training. This paper aims to investigate the potential of the object-level VLPM, Grounded Language-Image Pre-training (GLIP), for zero-shot nuclei detection. Specifically, we propose an innovative auto-prompting pipeline, named AttriPrompter, comprising attribute generation, attribute augmentation, and relevance sorting, to avoid subjective manual prompt design. AttriPrompter utilizes VLPMs' text-to-image alignment to create semantically rich text prompts, which are then fed into GLIP for initial zero-shot nuclei detection. Additionally, we propose a self-trained knowledge distillation framework, where GLIP serves as the teacher with its initial predictions used as pseudo labels, to address the challenges posed by high nuclei density, including missed detections, false positives, and overlapping instances. Our method exhibits remarkable performance in label-free nuclei detection, outperforming all existing unsupervised methods and demonstrating excellent generality. Notably, this work highlights the astonishing potential of VLPMs pre-trained on natural image-text pairs for downstream tasks in the medical field as well. Code will be released at https://github.com/wuyongjianCODE/AttriPrompter.

Figures (5)

• Figure 1: Method illustration. (a) AttriPrompter: utilizing GLIP li2022grounded and BLIP li2022blip for attribute generation; exploiting GPT model radford2019language for attribute augmentation; ending with relevance sorting of prompts composed of attribute words and medical nouns. (b) Iterative optimization through self-trained knowledge distillation. Panels (c) and (d) detail the attribute augmentation and relevance sorting processes.
• Figure 2: Positioning higher relevance prompts earlier in the sequence positively influences GLIP, enhancing prediction accuracy. The boxes are shown in white. The values of mAP are calculated across the entire MoNuSeg testing set.
• Figure 3: Semantic coverage of nuclei with diverse appearances after attribute augmentation. (a) Augmentation for color "purple". (b) Augmentation for shape "round". Adjectives within the same frame are synonyms, while those in different frames represent varying degrees of the attribute.
• Figure 4: Comparison output visualizations. The $1^{\text{st}} \sim 4^{\text{th}}$ rows: MoNuSeg, the $5^{\text{th}} \sim 8^{\text{th}}$ row: CoNSeP. Column: (a) WSPointA; (b) WSPPointA; (c) WSMixedA; (d) WNSeg; (e) SSNS; (f) PDAM; (g) DARCNN; (h) Freesolo; (i) SOP; (j) PSM; (k) CutLER; (l) VL-PLM; (m) VLDet; (n) MIU-VL; (o) VLPM-NuD; (p) ours; (q) fully-supervised YOLOX. Green, red, and yellow boxes denote true positives, false positives, and false negatives.
• Figure 5: Two cross-domain experiments: (a) the cross-domain capacity of Attriprompter, (b) the domain shift study for the whole system. "Dataset A" denotes the source domain, "Dataset B" denotes the target domain.