P2Object: Single Point Supervised Object Detection and Instance Segmentation

TL;DR

This work tackles the gap between fully supervised and point-supervised object detection and instance segmentation by proposing P2Object, a framework combining P2BNet (and P2BNet++) for generating high-quality, balanced pseudo boxes and P2MNet for pixel-level mask prediction. It introduces continuous object perception via SPSD sampling and boundary self-prediction to mitigate errors from discrete box sampling, enabling effective training of detectors and segmentors from single-point annotations. Empirical results across COCO, VOC, SBD, and Cityscapes show substantial gains over previous PSOD/WSIS methods and competitive performance with stronger backbones, illustrating the practical potential of low-cost point supervision to approach fully supervised performance. The approach provides a scalable, data-efficient path toward accurate object localization and segmentation with minimal annotation effort, and offers practical deployment advantages since inference relies on the trained detector/segmentor, with pseudo-labels used only during training.

Abstract

Object recognition using single-point supervision has attracted increasing attention recently. However, the performance gap compared with fully-supervised algorithms remains large. Previous works generated class-agnostic \textbf{\textit{proposals in an image}} offline and then treated mixed candidates as a single bag, putting a huge burden on multiple instance learning (MIL). In this paper, we introduce Point-to-Box Network (P2BNet), which constructs balanced \textbf{\textit{instance-level proposal bags}} by generating proposals in an anchor-like way and refining the proposals in a coarse-to-fine paradigm. Through further research, we find that the bag of proposals, either at the image level or the instance level, is established on discrete box sampling. This leads the pseudo box estimation into a sub-optimal solution, resulting in the truncation of object boundaries or the excessive inclusion of background. Hence, we conduct a series exploration of discrete-to-continuous optimization, yielding P2BNet++ and Point-to-Mask Network (P2MNet). P2BNet++ conducts an approximately continuous proposal sampling strategy by better utilizing spatial clues. P2MNet further introduces low-level image information to assist in pixel prediction, and a boundary self-prediction is designed to relieve the limitation of the estimated boxes. Benefiting from the continuous object-aware \textbf{\textit{pixel-level perception}}, P2MNet can generate more precise bounding boxes and generalize to segmentation tasks. Our method largely surpasses the previous methods in terms of the mean average precision on COCO, VOC, SBD, and Cityscapes, demonstrating great potential to bridge the performance gap compared with fully supervised tasks.

Figures (11)

• Figure 1: i) Left: The previous methods, e.g. UFO$^2$, treat all the candidate proposals (like object1, object2 and background) of the image in a single bag, and the numbers of proposals for different are unbalanced. ii) Middle: P2BNet constructs balanced instance-level proposal bags anchor-like sampling and refine the proposals in a coarse-to-fine fashion, which is beneficial for MIL optimization. iii) Right: However, the bag of proposals, either at the image level or instance level, is established on discrete box sampling, which leads to truncated or over-inclusive prediction at the object boundary. P2MNet aims to transfer the optimization to pixel-level and predicts continuous object masks. The estimated pseudo boxes and masks are used to supervise the detector and segmentor, which significantly improves the performance.
• Figure 2: With point annotation, P2BNet++ predicts the pseudo box (red), P2MNet estimates the object mask (blue) and the minimum circumscribed rectangle updates the pseudo box (yellow). Without BSP, the object mask truncates the object influenced by the original pseudo box (first line) or brings much background (second line). With BSP, the above problems are relieved. (Best viewed in color.)
• Figure 3: The general pipelines for the proposed CBP, P2BNet, P2BNet++ and P2MNet, respectively. 'I' is the input image, 'P' is the single point annotation, 'B' is the pseudo box, and 'M' is the pseudo mask. 'SP' is the sampling, 'H' is the MIL network, 'PE' is position embedding, and 'CP' is the convolution parameter. The 'cat' is the 'concat' operation, 'sup.' is the 'supervise' operation, and 'conv.' is the 'convolution' operation.
• Figure 4: The architecture of P2BNet. Firstly, proposal bags are fixedly sampled around point annotations for classifier training to predict coarse pseudo boxes in the CBP stage. Secondly, to predict refined pseudo boxes in the PBR stage, high-quality proposal bags and negative proposals are sampled with coarse pseudo boxes for training. Finally, the pseudo boxes train a classic detector. (Best viewed in colour.)
• Figure 5: Details of sampling strategies in the CBP and PBR stages. The arrows in PBR sampling mean the offset of center jitter. The basic sampling is conducted through the center jitter following scale and aspect ratio jitter. The SPSD sampling, adopting spatial distillation policy, is used in P2BNet++.