A comprehensive review and new taxonomy on superpixel segmentation

TL;DR

The paper addresses the fragmentation of superpixel segmentation literature by introducing a taxonomy based on processing steps and feature abstraction levels, enabling principled comparisons across a wide range of methods. It also provides a new benchmark that evaluates 23 superpixel methods (and a grid baseline) on nine criteria, supported by five diverse datasets, and analyzes performance across multiple metrics including BR, UE, EV, SIRS, and CO. Key findings reveal distinct trade-offs: boundary-evolution methods tend to be highly compact and fast but may underperform in boundary adherence; path-based and neighborhood-based methods often offer superior delineation and homogeneity with varying runtimes; deep-learning approaches show strong efficiency but heterogeneous quality and often require post-processing for connectivity. Overall, the taxonomy and benchmark offer a practical tool for selecting methods tailored to specific applications and guide future research toward tools that balance delineation, compactness, and runtime in real-world tasks.

Abstract

Superpixel segmentation consists of partitioning images into regions composed of similar and connected pixels. Its methods have been widely used in many computer vision applications since it allows for reducing the workload, removing redundant information, and preserving regions with meaningful features. Due to the rapid progress in this area, the literature fails to catch up on more recent works among the compared ones and to categorize the methods according to all existing strategies. This work fills this gap by presenting a comprehensive review with new taxonomy for superpixel segmentation, in which methods are classified according to their processing steps and processing levels of image features. We revisit the recent and popular literature according to our taxonomy and evaluate 20 strategies based on nine criteria: connectivity, compactness, delineation, control over the number of superpixels, color homogeneity, robustness, running time, stability, and visual quality. Our experiments show the trends of each approach in pixel clustering and discuss individual trade-offs. Finally, we provide a new benchmark for superpixel assessment, available at https://github.com/IMScience-PPGINF-PucMinas/superpixel-benchmark.

Figures (15)

• Figure 1: Superpixel segmentation examples, in which superpixel borders are shown in red. Although boundary adherence, regularity, and compactness are essential properties, (a) superpixels with higher regularity and compactness have poor boundary adherence. Conversely, (b) superpixel methods focused on boundary adherence may present irregular contours due to their sensitivity to subtle color variations.
• Figure 2: Categories of each processing step in superpixel taxonomy.
• Figure 3: The main processing categories in superpixel taxonomy and the methods that conform with each one.
• Figure 4: The usage of neural networks in superpixel segmentation. The color of each method relates to its main processing category color in Figure \ref{['fig:sunburst_diagram']}.
• Figure 5: Results for BR and UE on Birds+Insects+Sky+ECSSD and NYUV2 datasets.