PFPs: Prompt-guided Flexible Pathological Segmentation for Diverse Potential Outcomes Using Large Vision and Language Models

TL;DR

This work addresses the need for flexible segmentation targets in pathology by integrating language prompts from a lightweight LLM with spatial prompts into a segmentation backbone based on EfficientSAM. The method enables multi-class, multi-task kidney pathology segmentation, using LoRA-tuned TinyLlama embeddings and a dynamic head to adapt to various tasks and unseen cases. Key contributions include a practical pipeline combining free-text prompts with spatial cues, a 4-class, 9-task kidney dataset, and comprehensive comparisons of prompt strategies and training regimes, including complete versus incomplete training and LoRA fine-tuning. Results indicate that language-guided segmentation offers greater flexibility and generalization under certain settings, signaling potential for improved diagnostic quantification in renal pathology, albeit with limitations due to dataset size and resources.

Abstract

The Vision Foundation Model has recently gained attention in medical image analysis. Its zero-shot learning capabilities accelerate AI deployment and enhance the generalizability of clinical applications. However, segmenting pathological images presents a special focus on the flexibility of segmentation targets. For instance, a single click on a Whole Slide Image (WSI) could signify a cell, a functional unit, or layers, adding layers of complexity to the segmentation tasks. Current models primarily predict potential outcomes but lack the flexibility needed for physician input. In this paper, we explore the potential of enhancing segmentation model flexibility by introducing various task prompts through a Large Language Model (LLM) alongside traditional task tokens. Our contribution is in four-fold: (1) we construct a computational-efficient pipeline that uses finetuned language prompts to guide flexible multi-class segmentation; (2) We compare segmentation performance with fixed prompts against free-text; (3) We design a multi-task kidney pathology segmentation dataset and the corresponding various free-text prompts; and (4) We evaluate our approach on the kidney pathology dataset, assessing its capacity to new cases during inference.

Figures (3)

• Figure 1: Problem definition: For pathology images, the small, diverse structures and their complex relationships demand higher flexibility in image segmentation, which current segmentation methods may not meet. Sometimes, the segmentation target is ambiguous without the language prompt.
• Figure 2: Idea of our work: The addition of free text provides clarity for accurately describing the segmentation target compared with using the spatial annotation only. We simulate a multi-class and multi-task dataset of kidney pathology in this work. Multiple classes of units are essential in renal pathology image analysis, such as the proximal tubule, distal tubule, tuft, capsule, nuclei, etc. For each class, there can be different segmentation tasks. This figure shows the unit classes and tasks we prepared for our work. Our approach investigates the efficacy of controlling segmentation models through natural language prompts and point-based methods, highlighting the enhanced flexibility of these prompts in contrast to conventional fixed task IDs.
• Figure 3: Proposed pipeline: This figure presents the pipeline using free text and points as segmentation prompts. The lower part illustrates the segmentation backbone, while the upper part shows how the embeddings of free-text prompts are generated and integrated into the segmentation backbone in three stages. The trainable blocks and frozen blocks are highlighted with fire and ice icons, respectively.