DExTeR: Weakly Semi-Supervised Object Detection with Class and Instance Experts for Medical Imaging

TL;DR

DExTeR presents a transformer-based weakly semi-supervised object detector tailored for medical imaging by introducing class-guided MSDA, CLICK-MoE, and a multi-point training strategy to robustly convert point annotations into accurate bounding boxes. The method strengthens Point-to-Box regression within a DETR framework, decoupling class and instance information to improve discrimination among adjacent structures. Across Endoscapes, VinDr-CXR, and EUS-D130, DExTeR achieves state-of-the-art performance and translates into strong downstream gains for student detectors, reducing annotation costs while maintaining high localization accuracy. These advances offer a practical pathway to scalable, annotation-efficient landmark detection in challenging medical imaging domains.

Abstract

Detecting anatomical landmarks in medical imaging is essential for diagnosis and intervention guidance. However, object detection models rely on costly bounding box annotations, limiting scalability. Weakly Semi-Supervised Object Detection (WSSOD) with point annotations proposes annotating each instance with a single point, minimizing annotation time while preserving localization signals. A Point-to-Box teacher model, trained on a small box-labeled subset, converts these point annotations into pseudo-box labels to train a student detector. Yet, medical imagery presents unique challenges, including overlapping anatomy, variable object sizes, and elusive structures, which hinder accurate bounding box inference. To overcome these challenges, we introduce DExTeR (DETR with Experts), a transformer-based Point-to-Box regressor tailored for medical imaging. Built upon Point-DETR, DExTeR encodes single-point annotations as object queries, refining feature extraction with the proposed class-guided deformable attention, which guides attention sampling using point coordinates and class labels to capture class-specific characteristics. To improve discrimination in complex structures, it introduces CLICK-MoE (CLass, Instance, and Common Knowledge Mixture of Experts), decoupling class and instance representations to reduce confusion among adjacent or overlapping instances. Finally, we implement a multi-point training strategy which promotes prediction consistency across different point placements, improving robustness to annotation variability. DExTeR achieves state-of-the-art performance across three datasets spanning different medical domains (endoscopy, chest X-rays, and endoscopic ultrasound) highlighting its potential to reduce annotation costs while maintaining high detection accuracy.

Figures (7)

• Figure 1: Pipeline of the Weakly Semi-Supervised Object Detection with Points (WSSOD-P) framework. The framework aims to train a detector while minimizing box annotation time by predominantly using faster point annotations across the dataset. (a) Box annotations are used to sample points and train a Point-to-Box regressor. (b) The trained Point-to-Box model generates pseudo-box labels for point-annotated images. (c) A student detector is trained on both the box-labeled and pseudo-labeled images to improve detection performance.
• Figure 2: DExTeR Model: it leverages multiple point groups during training and incorporates class-guided Multi-Scale Deformable Attention (MSDA) alongside the Class, Instance, and Common Knowledge Mixture of Experts (CLICK-MoE). Point queries are iteratively refined using a Point-to-Box strategy implemented through stacked decoder blocks.
• Figure 3: Prompt encoding. a) Depending on the input type, either a point encoder or a box encoder is used. Each point $(x, y, c)$ or box $(x, y, w, h, c)$ is encoded using positional encoding and category embedding, then merged by a linear layer into the final representation. b) Point-to-Box refinement strategy: Encodes point annotations and predicts refinements after each decoder stage to form box-queries, which are re-encoded for subsequent refinement.
• Figure 4: Class, Instance, and Common Knowledge Mixture of Experts (CLICK-MoE): input object queries are processed through three specialized experts—a dynamically generated instance expert, a class-specific expert based on the query's label, and a common knowledge expert. The outputs from these experts are combined using weighted element-wise summation, where the weights are $W_{\text{instance}}$, $W_{\text{class}}$, and $W_{\text{gen}}$.
• Figure 5: Instance Expert weight generation. The object queries, along with a learnable embedding, undergo self-attention and are projected with a linear layer to produce instance-aware parameters for the instance experts.