PanoSAMic: Panoramic Image Segmentation from SAM Feature Encoding and Dual View Fusion

TL;DR

PanoSAMic addresses the challenge of panoramic semantic segmentation by reusing a frozen SAM encoder, expanding it with multi-stage feature outputs, a spatio-modal fusion module, and a dual-view panoramic decoder that uses spherical attention to mitigate edge distortions. The method introduces Moving CBAM for region-aware multi-modal fusion, horizontal positional encoding, and instance-guided semantic refinement to improve boundary accuracy and label consistency across RGB, depth, and normals inputs. It achieves state-of-the-art results on Stanford2D3DS and Matterport3D across RGB, RGB-D, and RGB-D-N modalities, while maintaining efficiency due to a frozen backbone and lightweight fusion/decoder blocks. The work demonstrates that SAM can be effectively repurposed for panoramic segmentation, enabling robust open- and closed-set performance with strong generalization and practical applicability in robotics and immersive vision tasks.

Abstract

Existing image foundation models are not optimized for spherical images having been trained primarily on perspective images. PanoSAMic integrates the pre-trained Segment Anything (SAM) encoder to make use of its extensive training and integrate it into a semantic segmentation model for panoramic images using multiple modalities. We modify the SAM encoder to output multi-stage features and introduce a novel spatio-modal fusion module that allows the model to select the relevant modalities and best features from each modality for different areas of the input. Furthermore, our semantic decoder uses spherical attention and dual view fusion to overcome the distortions and edge discontinuity often associated with panoramic images. PanoSAMic achieves state-of-the-art (SotA) results on Stanford2D3DS for RGB, RGB-D, and RGB-D-N modalities and on Matterport3D for RGB and RGB-D modalities. https://github.com/dfki-av/PanoSAMic

Figures (8)

• Figure 1: SAM kirillov2023segment is not trained for semantic segmentation and is unable to fully handle panoramic images (white is unsegmented). PanoSAMic uses the SAM pre-trained encoder and is tailored for semantic segmentation of panoramic images.
• Figure 2: PanoSAMic architecture. Two views of the same panoramic input are fed into the SAM encoder. The fusion block combines and refines the features from the processed input modalities. The features are then concatenated and passed into the decoder along with a horizontal positional encoding. The semantic decoder outputs the fused segmentation which is refined with mask prediction.
• Figure 3: Objects in panoramic images that are disconnected on the edges are processed as whole in the shifted view.
• Figure 4: Our novel blocks used for feature fusion and dual view fusion.
• Figure 5: Comparison of the qualitative segmentation results of our PanoSAMic model for different scenes from the Stanford2D3DS dataset armeni2017joint and using different input configurations.