Rethinking Features-Fused-Pyramid-Neck for Object Detection

TL;DR

This work tackles feature misalignment in FPN-like necks of real-time object detectors by proposing an unfused backbone architecture (Independent Hierarchy Pyramid, IHP) and a suite of alignment and efficiency techniques: Soft Nearest Neighbor Interpolation (SNI) to relax upsampling influence, Features Adaptive Selection in Extended Spatial Windows (ESD) to preserve spatial details during downsampling, and GSConvE enhancements for lightweight models. Together, these form the Secondary Features Alignment (SA) solution, enabling per-scale learning and reduced cross-level noise while maintaining speed. Empirical evaluation on Pascal VOC and MS COCO demonstrates consistent improvements across baselines, with SA-based detectors achieving state-of-the-art accuracy among real-time methods; ablation studies validate the contributions of IHP, SNI, ESD, and GSConvE. The approach offers practical benefits for edge devices by improving detection quality without prohibitive computational cost and provides a clear direction for further reducing feature misalignment in multi-scale detectors, with code to be released at the provided repository URL.

Abstract

Multi-head detectors typically employ a features-fused-pyramid-neck for multi-scale detection and are widely adopted in the industry. However, this approach faces feature misalignment when representations from different hierarchical levels of the feature pyramid are forcibly fused point-to-point. To address this issue, we designed an independent hierarchy pyramid (IHP) architecture to evaluate the effectiveness of the features-unfused-pyramid-neck for multi-head detectors. Subsequently, we introduced soft nearest neighbor interpolation (SNI) with a weight downscaling factor to mitigate the impact of feature fusion at different hierarchies while preserving key textures. Furthermore, we present a features adaptive selection method for down sampling in extended spatial windows (ESD) to retain spatial features and enhance lightweight convolutional techniques (GSConvE). These advancements culminate in our secondary features alignment solution (SA) for real-time detection, achieving state-of-the-art results on Pascal VOC and MS COCO. Code will be released at https://github.com/AlanLi1997/rethinking-fpn. This paper has been accepted by ECCV2024 and published on Springer Nature.

Figures (5)

• Figure 1: The IHP and seven other typical neck architectures. The three heat maps are come from the third (P3), fourth (P4), and fifth (P5) hierarchies of the FPN-neck model, respectively. The area of interest for each hierarchies' features is highlighted in red; the darker the color, the higher the feature weight. It is evident that lower-level hierarchies prioritize local features, while higher-level hierarchies focus more on global features. In other words, there is a representation bias at different hierarchical levels. If these features are combined by point-to-point fusion without first addressing the misalignment issue, it may introduce noise rather than enrich semantics.
• Figure 2: Illustrations of the features misalignment in fusion and the SNI. These numbers, 22-88, are just markers of different local-features not real feature values.
• Figure 3: A comparison of the SNI and traditional method. This result could be reproduced by using SNI source code (model:Yolov5n-panet). The same size scaling before and after up-sampling is for intuitive comparison.
• Figure 4: Illustrations of the ESD-I and ESD-II. It must be mentioned that although the size of the extended window does not have to be same in the ESD-I, pooling is a non-learnable operation, so the features from an unequal window may become noise for other and bring the trouble of the features misalignment again. The ESD-II is free on size of the extended window because of the the learnable fusion like the SPPNet sppnet for the YOLOv3.
• Figure 5: Illustrations of the GSConvE-I and II.