RFAssigner: A Generic Label Assignment Strategy for Dense Object Detection

TL;DR

RFAssigner addresses scale imbalance in dense object detection by combining an initial point-prior positive set with Gaussian Receptive Field (GRF) based supplementary positives, all within a differentiable framework. It models each ground-truth object and each feature point as 2D Gaussians and computes a receptive-field distance using the Gaussian Combined Distance (GCD), converting it to a similarity with $\text{RFD} = \exp(-\sqrt{D_{gc}^{2}(\mathcal{N}_{gt}, \mathcal{N}_{tr})})$. Ambiguous Matching further reassigns difficult unassigned samples to multiple GTs via a masked combination $M_{result}=M_{p}+M_{f}(1-M_{p})$, balancing positives without altering the negative center-prior. Across AI-TOD-v2, VisDrone-2019, and COCO-2017, RFAssigner demonstrates consistent improvements over prior label assignment methods, with RFAssigner* achieving strong cross-scale performance when leveraging finer FPN levels, all without increasing inference cost.

Abstract

Label assignment is a critical component in training dense object detectors. State-of-the-art methods typically assign each training sample a positive and a negative weight, optimizing the assignment scheme during training. However, these strategies often assign an insufficient number of positive samples to small objects, leading to a scale imbalance during training. To address this limitation, we introduce RFAssigner, a novel assignment strategy designed to enhance the multi-scale learning capabilities of dense detectors. RFAssigner first establishes an initial set of positive samples using a point-based prior. It then leverages a Gaussian Receptive Field (GRF) distance to measure the similarity between the GRFs of unassigned candidate locations and the ground-truth objects. Based on this metric, RFAssigner adaptively selects supplementary positive samples from the unassigned pool, promoting a more balanced learning process across object scales. Comprehensive experiments on three datasets with distinct object scale distributions validate the effectiveness and generalizability of our method. Notably, a single FCOS-ResNet-50 detector equipped with RFAssigner achieves state-of-the-art performance across all object scales, consistently outperforming existing strategies without requiring auxiliary modules or heuristics.

Figures (3)

• Figure 1: Comparison of label assignment pipelines. FCOS and RFLA define positive samples using a point prior and a GRF prior, respectively, with all other samples treated as negatives. RFAssigner synthesizes these approaches: it first initializes a positive set via a point prior (like FCOS), supplements this set using a GRF-based selection (inspired by RFLA), and finally assigns continuous positive and negative weights to all samples following the DW paradigm.
• Figure 2: Visualization of different label assignment strategies. (Left) FCOS uses a point prior, assigning all locations within the GT box as positive (green). (Center) RFLA uses a hierarchical assignment based on RFD scores, which can designate locations outside the GT box as positive. (Right) RFAssigner begins with a point prior (green) and then adaptively selects ambiguous samples (blue) based on RFD statistics. These ambiguous samples are dynamically matched to GTs, allowing the assignment to be optimized throughout training.
• Figure 3: Qualitative detection results on the AI-TOD-v2 validation set. From left to right: DW, DW* + RFLA, and our RFAssigner*. Ground-truth boxes are shown in green, and predictions are in red. At a high confidence threshold (e.g., 0.5), DW* + RFLA does not consistently improve upon DW. In contrast, RFAssigner* demonstrates markedly superior detection performance.