Accelerating Non-Maximum Suppression: A Graph Theory Perspective

TL;DR

This paper systematically analyzes NMS from a graph theory perspective for the first time, revealing its intrinsic structure, and proposes two optimization methods, namely QSI-NMS and BOE-NMS.

Abstract

Non-maximum suppression (NMS) is an indispensable post-processing step in object detection. With the continuous optimization of network models, NMS has become the ``last mile'' to enhance the efficiency of object detection. This paper systematically analyzes NMS from a graph theory perspective for the first time, revealing its intrinsic structure. Consequently, we propose two optimization methods, namely QSI-NMS and BOE-NMS. The former is a fast recursive divide-and-conquer algorithm with negligible mAP loss, and its extended version (eQSI-NMS) achieves optimal complexity of $\mathcal{O}(n\log n)$. The latter, concentrating on the locality of NMS, achieves an optimization at a constant level without an mAP loss penalty. Moreover, to facilitate rapid evaluation of NMS methods for researchers, we introduce NMS-Bench, the first benchmark designed to comprehensively assess various NMS methods. Taking the YOLOv8-N model on MS COCO 2017 as the benchmark setup, our method QSI-NMS provides $6.2\times$ speed of original NMS on the benchmark, with a $0.1\%$ decrease in mAP. The optimal eQSI-NMS, with only a $0.3\%$ mAP decrease, achieves $10.7\times$ speed. Meanwhile, BOE-NMS exhibits $5.1\times$ speed with no compromise in mAP.

Figures (9)

• Figure 1: Dynamic programming in topological sorting. The color of the node represents the $\mathit{\delta}$ value, i.e., black represents $1$, and white represents $0$. Before suppression, each node is black. In topological sorting, traversed arcs are represented by dashed lines, showing they have been removed from the graph. After the topological sorting is completed, we can find that nodes $1$, $6$, and $8$ are all black, that is, the last boxes retained are $b_1$, $b_6$, and $b_8$.
• Figure 2: Statistical characteristics of graph $\mathcal{G}$ on MS COCO 2017 validation. \ref{['subfig:1a']} The scatter plot of $5000$$\mathcal{G}$s on MS COCO 2017. It indicates that the number of arcs $\vert \mathcal{E}\vert$ and the number of WCCs $\vert\mathcal{W}\vert$ exhibit an approximately linear relationship with the number of nodes $\vert\mathcal{V}\vert$, respectively. \ref{['subfig:1b']} The violin plot of the sizes of WCCs across different categories on MS COCO 2017. It reveals the distributional characteristics of the sizes of the WCCs. It shows that over $50\%$ of the WCCs have a size less than $5$, and more than $75\%$ have a size less than $10$.
• Figure 3: The key ideas behind QSI-NMS (left) and BOE-NMS (right). $\mathcal{G}$ (middle) contains many small weakly connected components (WCCs). QSI-NMS considers the global structure of the graph $\mathcal{G}$, where there are many WCCs. It selects a pivot (the red node on the left) and computes IOUs (orange edges) with all current subproblem nodes using a divide-and-conquer algorithm. BOE-NMS focuses on the local structure (the red dashed box) of $\mathcal{G}$, where most WCCs are quite small in size. It selects a node (the red node on the right) and only computes IOUs (orange edges) with its nearby nodes (solid arrows), which is derived from 2D plane geometric analysis (dashed arrows).
• Figure 4: A Cartesian tree for $B$. The x-axis represents the centroid, where the node on the left $\preceq_{\mathcal{C}}$ the one on the right. The y-axis represents the confidence score, where the node below $\preceq_{\mathcal{P}}$ the one above. The values of the sequence below the x-axis are the confidence scores of $B$.
• Figure 5: Four positions of $b^*$ relative to $b$.

Theorems & Definitions (29)

• Definition 1
• Proposition 1
• Theorem 1
• Corollary 1
• Definition 2
• Theorem 2
• Theorem 3
• Theorem 4
• Definition 3: Directed acyclic graph
• Definition 4: Strongly connected