Deep Spectral Improvement for Unsupervised Image Instance Segmentation

TL;DR

This work enhances deep spectral methods for unsupervised image instance segmentation by identifying that not all self-supervised backbone channels are informative. It introduces two channel-reduction modules, Noise Channel Reduction (NCR) based on entropy and Deviation-based Channel Reduction (DCR) based on standard deviation, to retain channels most useful for segmentation. It also replaces the conventional dot product with Bray-Curtis over Chebyshev (BoC) to form a more robust affinity matrix that captures both feature distributions and values. The approach yields improved mean IoU for instance segmentation on YouTube-VIS 2019 and OVIS datasets, with ablations showing strong synergy when NCR, DCR, and BoC are combined. Overall, the method advances unsupervised instance segmentation by refining feature selection and affinity construction to produce clearer eigensegments.

Abstract

Deep spectral methods reframe the image decomposition process as a graph partitioning task by extracting features using self-supervised learning and utilizing the Laplacian of the affinity matrix to obtain eigensegments. However, instance segmentation has received less attention compared to other tasks within the context of deep spectral methods. This paper addresses the fact that not all channels of the feature map extracted from a self-supervised backbone contain sufficient information for instance segmentation purposes. In fact, Some channels are noisy and hinder the accuracy of the task. To overcome this issue, this paper proposes two channel reduction modules: Noise Channel Reduction (NCR) and Deviation-based Channel Reduction (DCR). The NCR retains channels with lower entropy, as they are less likely to be noisy, while DCR prunes channels with low standard deviation, as they lack sufficient information for effective instance segmentation. Furthermore, the paper demonstrates that the dot product, commonly used in deep spectral methods, is not suitable for instance segmentation due to its sensitivity to feature map values, potentially leading to incorrect instance segments. A new similarity metric called Bray-Curtis over Chebyshev (BoC) is proposed to address this issue. It takes into account the distribution of features in addition to their values, providing a more robust similarity measure for instance segmentation. Quantitative and qualitative results on the Youtube-VIS2019 dataset highlight the improvements achieved by the proposed channel reduction methods and the use of BoC instead of the conventional dot product for creating the affinity matrix. These improvements are observed in terms of mean Intersection over Union and extracted instance segments, demonstrating enhanced instance segmentation performance. The code is available on: https://github.com/farnooshar/SpecUnIIS

Figures (14)

• Figure 1: Features extracted from the self-supervised backbone. (a) Input image. (b) A channel suitable for foreground-background segmentation. (c) A random channel which contains no valuable information. (d) A channel with the potential for instance segmentation. (e) A channel proper for instance segmentation, multiplied by the foreground mask. As can be seen, instances in the image are distinguishable by their pixel value.
• Figure 2: Potential appropriateness of channels for intense segmentation. As depicted in this figure, a specific channel shows promising potential for performing well in the instance segmentation task. These results were analyzed across different instances.
• Figure 3: Pipelines of deep spectral methods. (a) Workflow ofmelas2022deep. An affinity matrix is created using a dot product with features from a self-supervised backbone. Eigenvectors of the Laplacian matrix derived from the affinity matrix are utilized for segmentation tasks. (b) Application of the proposed NCR module on the features from the self-supervised backbone to remove noisy channels. The Fiedler eigenvector is then employed for foreground-background segmentation. (c) Pipeline for instance segmentation. Stable feature map channels are further reduced based on their standard deviation to enhance feature richness. The resulting feature map is multiplied by the foreground mask, and the affinity matrix is created using the BoC metric. Finally, pixels are clustered using the eigenvectors of the Laplacian matrix, resulting in instance segmentation.
• Figure 4: Visualization of some channels from the self-supervised backbone. As evident in this figure, lower entropy corresponds to a better representation of objects in images, while higher entropy results in a more unclear representation resembling noise.
• Figure 5: Influence of DCR on the distinction between instances in YouTube-VIS 2019 dataset. As depicted by the orange curve, the standard deviation of the channels diminishes while simultaneously, $\Delta$, which represents the average difference between instances, also decreases. Consequently, channels with a higher standard deviation will probably display a more significant average difference between instances.