Sparse Semi-DETR: Sparse Learnable Queries for Semi-Supervised Object Detection

TL;DR

This work tackles the limitations of DETR-based semi-supervised object detection arising from pseudo-label quality and overlapping predictions. It introduces Sparse Semi-DETR, a dual-network (student/teacher) framework that integrates a Query Refinement Module to produce higher-quality, fewer object queries and a Reliable Pseudo-Label Filtering Module to prune low-quality pseudo-labels. The approach yields significant gains on MS-COCO and Pascal VOC, notably improving detection of small and occluded objects and providing NMS-free, end-to-end learning. The findings demonstrate that refining queries and selectively filtering pseudo-labels can substantially boost semi-supervised DETR performance, suggesting practical benefits for scalable object detection with limited labeled data.

Abstract

In this paper, we address the limitations of the DETR-based semi-supervised object detection (SSOD) framework, particularly focusing on the challenges posed by the quality of object queries. In DETR-based SSOD, the one-to-one assignment strategy provides inaccurate pseudo-labels, while the one-to-many assignments strategy leads to overlapping predictions. These issues compromise training efficiency and degrade model performance, especially in detecting small or occluded objects. We introduce Sparse Semi-DETR, a novel transformer-based, end-to-end semi-supervised object detection solution to overcome these challenges. Sparse Semi-DETR incorporates a Query Refinement Module to enhance the quality of object queries, significantly improving detection capabilities for small and partially obscured objects. Additionally, we integrate a Reliable Pseudo-Label Filtering Module that selectively filters high-quality pseudo-labels, thereby enhancing detection accuracy and consistency. On the MS-COCO and Pascal VOC object detection benchmarks, Sparse Semi-DETR achieves a significant improvement over current state-of-the-art methods that highlight Sparse Semi-DETR's effectiveness in semi-supervised object detection, particularly in challenging scenarios involving small or partially obscured objects.

Figures (9)

• Figure 1: (a)-(b) Comparative Overview of SSOD advancements: Sparse Semi-DETR's major improvement lies in its Query Refinement Module and Reliable Pseudo-label Filtering Module, significantly enhancing detection of small or obscured objects and reliability in complex scenarios, surpassing all other methods as shown in the graph in (c).
• Figure 2: An overview of the Sparse Semi-DETR framework. It contains two networks: the student network and the teacher network. Labeled data is used for student network training, employing a supervised loss. Unlabeled data is fed to the teacher network with weak augmentation and the student network with strong augmentation. The teacher network takes unlabeled data to generate pseudo-labels. Here, the query refinement module provides refined queries to avoid incorrect bipartite matching with teacher-generated pseudo-labels. For a detailed overview of the query refinement module, see Figure \ref{['fig:queries']}. Furthermore, a Reliable Pseudo-label Filtering strategy is employed to filter low-quality pseudo-labels progressively during training.
• Figure 3: Overview of the Query Refinement Module. The query features from strong and weak augmented unlabeled images are refined through the Query Refinement Module. It amplifies the semantic representation of object queries and improves performance for small objects. For the best view, zoom in.
• Figure 4: Visual comparison of Sparse Semi-DETR with the two previous approaches on the COCO 10% label dataset. These results highlight Sparse Semi-DETR's capabilities, particularly in identifying small objects and those obscured by obstacles (as indicated by white arrows) in the third-row images. For optimal clarity and detail, please zoom in.
• Figure 5: The top-row figures display Semi-DETR's detection results, and the bottom-row shows Sparse Semi-DETR's outcomes. Both networks were trained for 120k iterations using one-to-many and one-to-one strategies. Sparse Semi-DETR eliminates redundant bounding boxes in the bottom-left image and detects small objects, like knives, in the top-right image, as indicated by white arrows.