From Pixels to Objects: A Hierarchical Approach for Part and Object Segmentation Using Local and Global Aggregation

TL;DR

A hierarchical transformer-based model designed for sophisticated image segmentation tasks, effectively bridging the granularity of part segmentation with the comprehensive scope of object segmentation with the previous state-of-the-art.

Abstract

In this paper, we introduce a hierarchical transformer-based model designed for sophisticated image segmentation tasks, effectively bridging the granularity of part segmentation with the comprehensive scope of object segmentation. At the heart of our approach is a multi-level representation strategy, which systematically advances from individual pixels to superpixels, and ultimately to cohesive group formations. This architecture is underpinned by two pivotal aggregation strategies: local aggregation and global aggregation. Local aggregation is employed to form superpixels, leveraging the inherent redundancy of the image data to produce segments closely aligned with specific parts of the object, guided by object-level supervision. In contrast, global aggregation interlinks these superpixels, organizing them into larger groups that correlate with entire objects and benefit from part-level supervision. This dual aggregation framework ensures a versatile adaptation to varying supervision inputs while maintaining computational efficiency. Our methodology notably improves the balance between adaptability across different supervision modalities and computational manageability, culminating in significant enhancement in segmentation performance. When tested on the PartImageNet dataset, our model achieves a substantial increase, outperforming the previous state-of-the-art by 2.8% and 0.8% in mIoU scores for part and object segmentation, respectively. Similarly, on the Pascal Part dataset, it records performance enhancements of 1.5% and 2.0% for part and object segmentation, respectively.

Figures (7)

• Figure 1: Conceptual illustrations of LGFormer. On the left, LGFormer follows a hierarchical aggregation pathway, elevating features from pixels to parts to objects. The right figure shows the model's ability to progressively restore segmentation predictions from the object level to the original image resolution.
• Figure 2: Overview of LGFormer. Pixel-level features are extracted by a light convolution stem. In the initial ViT stages, these features are refined into part-level superpixels via Superpixel Context Aggregation (SCA). In deeper ViT layers, superpixels are aggregated into object-level groups using Group Context Aggregation (GCA).
• Figure 3: Illustration of interactions and spatial relationships among three hierarchical levels: pixels, superpixels, and groups. For clarity, the illustration presents a scenario involving only a single superpixel and a single group token. (a) Transitioning from pixels to superpixels involves iterative refinement of superpixels on a local scale through SCA. (b) Advancing from superpixels to groups, the refinement of groups on a global scale is facilitated by GCA.
• Figure 4: Enhanced Detail with Association-Aware Upsampling. In contrast to the conventional bilinear upsampled feature map, our association-aware upsampled feature map achieves sharper boundary delineation and retains greater semantic detail, which is crucial for detailed segmentation tasks.
• Figure 5: Visualization of Part and Object Semantic Emergence. (a) In object segmentation, superpixels reveal emerging part semantics. (b) Conversely, during part segmentation, object semantics become apparent within groups.