LesionLocator: Zero-Shot Universal Tumor Segmentation and Tracking in 3D Whole-Body Imaging

TL;DR

LesionLocator advances automated, zero-shot 3D segmentation and longitudinal tracking of tumors in whole-body imaging by introducing a unified 4D promptable framework. It combines a densely promptable 3D UNet backbone with a GradICON-based prompt propagation module and trains on a large-scale mixture of real and synthetic longitudinal data to enable end-to-end segmentation and tracking across timepoints. The approach achieves human-level performance in zero-shot lesion segmentation, outperforms existing promptable baselines by about 10 Dice points, and delivers state-of-the-art tracking metrics, all while releasing synthetic data and model weights to the research community. This work has the potential to reduce radiologist workload and enable scalable, dataset-agnostic lesion monitoring across diverse body regions and imaging modalities in clinical practice.

Abstract

In this work, we present LesionLocator, a framework for zero-shot longitudinal lesion tracking and segmentation in 3D medical imaging, establishing the first end-to-end model capable of 4D tracking with dense spatial prompts. Our model leverages an extensive dataset of 23,262 annotated medical scans, as well as synthesized longitudinal data across diverse lesion types. The diversity and scale of our dataset significantly enhances model generalizability to real-world medical imaging challenges and addresses key limitations in longitudinal data availability. LesionLocator outperforms all existing promptable models in lesion segmentation by nearly 10 dice points, reaching human-level performance, and achieves state-of-the-art results in lesion tracking, with superior lesion retrieval and segmentation accuracy. LesionLocator not only sets a new benchmark in universal promptable lesion segmentation and automated longitudinal lesion tracking but also provides the first open-access solution of its kind, releasing our synthetic 4D dataset and model to the community, empowering future advancements in medical imaging. Code is available at: www.github.com/MIC-DKFZ/LesionLocator

Figures (8)

• Figure 1: Task Overview. This figure illustrates the core setup and challenges of our task: enabling accurate segmentation and tracking of tumors across multiple follow-up scans from a single initial prompt. This task is inherently difficult due to variability in patient positioning, imaging window, scan protocols, and irregular intervals between imaging sessions. By allowing the user to simply mark a tumor in the initial scan, our approach automates consistent segmentation and tracking across all subsequent scans, streamlining tumor burden assessment and disease progression monitoring.
• Figure 2: Overview of the Proposed Lesion Tracking Pipeline. Our model receives user-provided prompts on the patient's initial scan, which are then propagated through time by the Prompt Propagation Module, enabling accurate lesion tracking across timepoints. Lesion delineation is performed by passing these propagated prompts to the Segmentation Module. Crucially, we introduce the propagation of predicted masks from the previous scan as prompt for the current scan, allowing for autoregressive segmentation throughout time series.
• Figure 3: Dataset Statistics. (a) Overview of the pretraining dataset modality composition. (b) Distribution of lesion sizes and types in the fine-tuning dataset.
• Figure 4: Examples from the synthetic dataset used to augment the training process for lesion tracking. The dataset simulates disease progression through random lesion progression, based on anatomy-informed transformations and image augmentations. Additional examples are provided in the Appendix.
• Figure 5: Consistent Lesion Tracking Performance Over Time. For the baseline scan the initial Dice score distribution is shown using the LesionLocator segmentation model with box prompts. For follow-up scans, tracking is performed autoregressively, as proposed, using prior masks as prompts. Tracking accuracy relative to the baseline is measured as CPM@25 (lesion matches within 25mm) with corresponding Dice@25 (Dice score of matched lesions). The Dice for matched lesions remains consistently high, with matching accuracy above 80% and only a slight decrease over time. Note: Only a single patient in our real longitudinal dataset has a Follow-Up 3 scan, so this distribution is based on one scan with 4 lesions, of which 3 were correctly matched.