Opportunistic Promptable Segmentation: Leveraging Routine Radiological Annotations to Guide 3D CT Lesion Segmentation

TL;DR

This work tackles the scarcity of large-scale 3D CT lesion segmentations by leveraging existing radiologist GSPS annotations through Opportunistic Promptable Segmentation (OPS). It introduces SAM2CT, an extension of SAM2 with Memory-Conditioned Memories (MCM) and an extended prompt encoder to interpret line and arrow prompts, converting sparse annotations into 3D segmentations. On public lesion benchmarks, SAM2CT achieves high Dice scores (e.g., up to $DSC=0.803$ for mask prompts and $0.757$ for line prompts) and the lowest RECIST error, while real-world PACS data show 62% of SAM2CT masks are clinically acceptable without edits; ED cases reveal strong generalization. The results demonstrate that mining historical GSPS annotations can yield scalable, high-quality 3D CT segmentation datasets and enable broader radiology-grounded AI applications.

Abstract

The development of machine learning models for CT imaging depends on the availability of large, high-quality, and diverse annotated datasets. Although large volumes of CT images and reports are readily available in clinical picture archiving and communication systems (PACS), 3D segmentations of critical findings are costly to obtain, typically requiring extensive manual annotation by radiologists. On the other hand, it is common for radiologists to provide limited annotations of findings during routine reads, such as line measurements and arrows, that are often stored in PACS as GSPS objects. We posit that these sparse annotations can be extracted along with CT volumes and converted into 3D segmentations using promptable segmentation models, a paradigm we term Opportunistic Promptable Segmentation. To enable this paradigm, we propose SAM2CT, the first promptable segmentation model designed to convert radiologist annotations into 3D segmentations in CT volumes. SAM2CT builds upon SAM2 by extending the prompt encoder to support arrow and line inputs and by introducing Memory-Conditioned Memories (MCM), a memory encoding strategy tailored to 3D medical volumes. On public lesion segmentation benchmarks, SAM2CT outperforms existing promptable segmentation models and similarly trained baselines, achieving Dice similarity coefficients of 0.649 for arrow prompts and 0.757 for line prompts. Applying the model to pre-existing GSPS annotations from a clinical PACS (N = 60), SAM2CT generates 3D segmentations that are clinically acceptable or require only minor adjustments in 87% of cases, as scored by radiologists. Additionally, SAM2CT demonstrates strong zero-shot performance on select Emergency Department findings. These results suggest that large-scale mining of historical GSPS annotations represents a promising and scalable approach for generating 3D CT segmentation datasets.

Figures (5)

• Figure 1: (a) The standard interactive segmentation pipeline. A radiologist provides a prompt, which the promptable segmentation model converts to a 3D mask. Optionally, the radiologist can review this mask and provide corrections or additional prompts; (b) Our proposed opportunistic promptable segmentation pipeline. Pre-existing GSPS annotations found in PACS are converted into 3D masks using a promptable segmentation model. This can be done automatically over large PACS databases to create large-scale 3D segmentation datasets.
• Figure 2: Overview of SAM2CT Architecture. The blue arrow illustrates the standard SAM2 architecture where image features are fed directly from the image encoder to the memory encoder. The yellow arrow shows the memory-conditioned memories (MCM) of SAM2CT where the memory encoder receives image features after they have undergone memory attention.
• Figure 3: Visualization of SAM2CT produced contours for oncology cases. Original GSPS prompts (arrow, line measurement, and major-minor measurements) are shown alongside SAM2CT segmentations in axial, coronal, and sagittal views.
• Figure 4: Performance of models with test-time memory-conditioned-memory (MCM) model architecture adaptation. Results are shown for the hold-out test set (top) and the external test set (bottom).
• Figure 5: 2D Dice scores on hold-out test set as a function of out-of-plane distance from the original prompt. Slice index 0 corresponds to the prompted slice.