A Sparse Graph Formulation for Efficient Spectral Image Segmentation

TL;DR

The paper tackles the scalability and interpretability limitations of spectral image segmentation via Normalized Cuts by introducing a sparse graph that augments a grid with a small set of color-nodes. This yields a tractable NCut formulation where the energy decomposes into a color-mismatch term and a spatial-boundary term, with a single interpretable parameter $\mu$ controlling border detail. Empirically, the method achieves fast runtimes and strong segmentation quality, often outperforming traditional spectral methods and rivaling modern unsupervised approaches, especially when using DINO features. The results demonstrate the practical value of a sparse, interpretable spectral formulation for high-resolution images and transformer-derived features, offering a compelling alternative to dense, hard-to-interpret NCut variants.

Abstract

Spectral Clustering is one of the most traditional methods to solve segmentation problems. Based on Normalized Cuts, it aims at partitioning an image using an objective function defined by a graph. Despite their mathematical attractiveness, spectral approaches are traditionally neglected by the scientific community due to their practical issues and underperformance. In this paper, we adopt a sparse graph formulation based on the inclusion of extra nodes to a simple grid graph. While the grid encodes the pixel spatial disposition, the extra nodes account for the pixel color data. Applying the original Normalized Cuts algorithm to this graph leads to a simple and scalable method for spectral image segmentation, with an interpretable solution. Our experiments also demonstrate that our proposed methodology over performs both traditional and modern unsupervised algorithms for segmentation in both real and synthetic data.

Figures (4)

• Figure 1: Sample qualitative results on synthetic images. Top: Gaussian noise, $\sigma_g^2 = 1.5$. Bottom: Salt & Pepper, $d = 0.7$.
• Figure 2: Segmentation results for varying image sizes. Left: a sample image for $\ell = 500$. Right: runtime and mIoU results varying the number of image pixels $n = \ell^2$.
• Figure 3: Qualitative results on BSDS500 images for various $\mu$ using pixel RGB features. The proposed methods is performs fast image segmentation in various imaging settings despite its spectral nature. Its sole parameter ($\mu$) regulates the level of border detail in the final segmentation. (b)-(d) show the eigenvector found and (left) and the resp. segmentation (right). Both images are of size $481\times 321$ and the average runtime is shown in parentheses.
• Figure 4: Qualitative comparison of our method to other deep learning-based unsupervised segmentation methods on challenging BSDS500 images. Our proposed methodology is generally able to preserve borders well and, when using DINO features, qualitatively outperform other deep-learning based unsupervised segmentation techniques.