Acquiring Weak Annotations for Tumor Localization in Temporal and Volumetric Data

TL;DR

This research proposes an efficient annotation strategy for tumor detection and localization that is less accurate than per-pixel annotations but useful for creating large-scale datasets for screening tumors in various medical modalities.

Abstract

Creating large-scale and well-annotated datasets to train AI algorithms is crucial for automated tumor detection and localization. However, with limited resources, it is challenging to determine the best type of annotations when annotating massive amounts of unlabeled data. To address this issue, we focus on polyps in colonoscopy videos and pancreatic tumors in abdominal CT scans; both applications require significant effort and time for pixel-wise annotation due to the high dimensional nature of the data, involving either temporary or spatial dimensions. In this paper, we develop a new annotation strategy, termed Drag&Drop, which simplifies the annotation process to drag and drop. This annotation strategy is more efficient, particularly for temporal and volumetric imaging, than other types of weak annotations, such as per-pixel, bounding boxes, scribbles, ellipses, and points. Furthermore, to exploit our Drag&Drop annotations, we develop a novel weakly supervised learning method based on the watershed algorithm. Experimental results show that our method achieves better detection and localization performance than alternative weak annotations and, more importantly, achieves similar performance to that trained on detailed per-pixel annotations. Interestingly, we find that, with limited resources, allocating weak annotations from a diverse patient population can foster models more robust to unseen images than allocating per-pixel annotations for a small set of images. In summary, this research proposes an efficient annotation strategy for tumor detection and localization that is less accurate than per-pixel annotations but useful for creating large-scale datasets for screening tumors in various medical modalities.

Figures (4)

• Figure 1: Annotation and propagation process of the proposed Drag&Drop. The pancreas and tumor are represented by the colors orange and green, respectively. Given the weak label, we expand it to a lesion marker (teal green) and background markers (red). We then utilize the marker-based watershed algorithm to generate the initial segmentation area (blue), following which dilated tumors (pink) are applied to compute the masked back-propagation.
• Figure 2: Illustration of Drag&Drop annotations on the JHH and SUN-SEG datasets.
• Figure 3: FROC curves of nnUNet trained by Drag&Drop and per-pixel annotation strategy with a certain budget on the JHH dataset.
• Figure 4: Illustration of the segmentation result of SAM using Drag&Drop and bounding box as a prompt.