How to Efficiently Annotate Images for Best-Performing Deep Learning Based Segmentation Models: An Empirical Study with Weak and Noisy Annotations and Segment Anything Model

TL;DR

This study investigates the cost-effectiveness of weak and noisy image annotations for training deep segmentation models across four datasets (VOC2012, WATER, LiTS17, BraTS20). By combining a three-step methodology—annotation time/quality estimation, programmatic imperfect annotation generation, and annotation-budget-restricted training—with four model-dataset pairings, the authors show that precise pixel-level labels are rarely the best use of a limited labeling budget. They demonstrate that noisy and weak annotations, particularly when paired with tools like the Segment Anything Model (SAM) or CRF-based regularization, can achieve comparable $mIoU$ performance at substantially lower cost, with the best option depending on the domain and annotation style. The work provides practical guidance for budget-conscious annotation strategies and introduces a framework and codebase for evaluating annotation-cost trade-offs across segmentation tasks.

Abstract

Deep neural networks (DNNs) have demonstrated exceptional performance across various image segmentation tasks. However, the process of preparing datasets for training segmentation DNNs is both labor-intensive and costly, as it typically requires pixel-level annotations for each object of interest. To mitigate this challenge, alternative approaches such as using weak labels (e.g., bounding boxes or scribbles) or less precise (noisy) annotations can be employed. Noisy and weak labels are significantly quicker to generate, allowing for more annotated images within the same time frame. However, the potential decrease in annotation quality may adversely impact the segmentation performance of the resulting model. In this study, we conducted a comprehensive cost-effectiveness evaluation on six variants of annotation strategies (9~10 sub-variants in total) across 4 datasets and conclude that the common practice of precisely outlining objects of interest is virtually never the optimal approach when annotation budget is limited. Both noisy and weak annotations showed usage cases that yield similar performance to the perfectly annotated counterpart, yet had significantly better cost-effectiveness. We hope our findings will help researchers be aware of the different available options and use their annotation budgets more efficiently, especially in cases where accurately acquiring labels for target objects is particularly costly. Our code will be made available on https://github.com/yzluka/AnnotationEfficiency2D.

Figures (7)

• Figure 1: Computer-generated annotations that assimilate the six different annotation styles on VOC2012. Blue contour indicates precise annotation, red contour/scribbles/point/bounding boxes indicate target objects, while the other color indicate background.
• Figure 2: Effects of Training Mask Quality on Model Performance with Fixed Sample Counts. To ensure the robustness of this trend, we experiment with the consumption of varying fractions of total costs or budgets, represented as different numbers of annotated images. The dotted line represents general trends for easier visual interpretation and does not correspond to a linear regression. The p-value is calculated using Spearman's rank correlation coefficient.
• Figure 3: The two plots show the approximated cost-effectiveness of different annotation styles for dataset in natural domain. The annotation styles positioned closer to the upper-left corners have higher cost-effectiveness.
• Figure 4: The two plots show the approximated cost-effectiveness of different annotation styles for dataset in CT and MRI. both SAM and MedSAM MedSAM are included as candidate model for processing bounding box annotations.
• Figure 5: Computer-generated annotations that assimilate the six different annotation styles on WATER.