Weakly Supervised Pixel-Level Annotation with Visual Interpretability

TL;DR

This paper tackles the challenge of producing accurate pixel-level annotations from image-level labels in medical imaging while maintaining interpretability and reliability. It presents the Auto Annotation eXplainable (AXX) Model, an ensemble of ResNet50, EfficientNet, and DenseNet augmented with XGrad-CAM and Monte Carlo Dropout, which outputs pixel-level masks for diseased regions only when predictions are confident and consistent across models. The approach combines three key contributions: (i) ensemble-based robust classification, (ii) weakly supervised pixel-level annotation via intersected saliency maps, and (iii) uncertainty quantification for open-set detection that flags ambiguous cases for expert review. Experimental results on TBX11K and Fire datasets show state-of-the-art accuracy and IoU, alongside meaningful uncertainty signaling, demonstrating strong generalizability and potential for clinical deployment and other image-analysis tasks.

Abstract

Medical image annotation is essential for diagnosing diseases, yet manual annotation is time-consuming, costly, and prone to variability among experts. To address these challenges, we propose an automated explainable annotation system that integrates ensemble learning, visual explainability, and uncertainty quantification. Our approach combines three pre-trained deep learning models - ResNet50, EfficientNet, and DenseNet - enhanced with XGrad-CAM for visual explanations and Monte Carlo Dropout for uncertainty quantification. This ensemble mimics the consensus of multiple radiologists by intersecting saliency maps from models that agree on the diagnosis while uncertain predictions are flagged for human review. We evaluated our system using the TBX11K medical imaging dataset and a Fire segmentation dataset, demonstrating its robustness across different domains. Experimental results show that our method outperforms baseline models, achieving 93.04% accuracy on TBX11K and 96.4% accuracy on the Fire dataset. Moreover, our model produces precise pixel-level annotations despite being trained with only image-level labels, achieving Intersection over Union IoU scores of 36.07% and 64.7%, respectively. By enhancing the accuracy and interpretability of image annotations, our approach offers a reliable and transparent solution for medical diagnostics and other image analysis tasks.

Figures (3)

• Figure 1: Architecture of the Auto Annotation eXplainable (AAX) Model. The pipeline integrates ResNet50, EfficientNet, and DenseNet with Monte Carlo Dropout and XGrad-CAM. Images with high uncertainty in at least two models are flagged for expert review, while confident predictions are classified as "Healthy" or "Diseased." For diseased cases, intersected saliency maps generate a binary mask highlighting disease-specific regions.
• Figure 2: Comparative Results of AAX Model and Baseline Methods on Fire Dataset. Visual comparison of input images, ground truth masks, and model outputs from DenseNet, EfficientNet, ResNet50, and the proposed AAX model. Columns show XGrad-CAM saliency maps and binary masks for each model. The AAX model’s binary masks align more closely with ground truth annotations, demonstrating improved feature localization and pixel-level annotation accuracy.
• Figure 3: Comparative Results of AAX Model and Baseline Methods on TBX11K Dataset. Visual comparison of chest X-ray inputs, ground truth annotations (bounding boxes and binary masks), and model outputs from DenseNet, EfficientNet, ResNet50, and the proposed AAX model. The AAX model's binary masks align more closely with ground truth annotations, demonstrating improved pixel-level localization despite being trained using only image-level labels.