iShape: A First Step Towards Irregular Shape Instance Segmentation

TL;DR

This work introduces iShape, a dataset tailored for irregular-shape instance segmentation, featuring six sub-datasets that stress overlaps, aspect-ratio extremes, and numerous connected components. It shows that existing methods struggle on iShape and introduces ASIS, an affinity-based baseline that mimics human contour-following through perception-and-reasoning, using a dataset-adaptive affinity kernel. Through extensive experiments and ablations, the paper demonstrates ASIS's superior performance on iShape and provides insights into kernel design and training strategies. The dataset and code are released to promote research on irregular and arbitrary shape segmentation beyond traditional benchmarks.

Abstract

In this paper, we introduce a brand new dataset to promote the study of instance segmentation for objects with irregular shapes. Our key observation is that though irregularly shaped objects widely exist in daily life and industrial scenarios, they received little attention in the instance segmentation field due to the lack of corresponding datasets. To fill this gap, we propose iShape, an irregular shape dataset for instance segmentation. iShape contains six sub-datasets with one real and five synthetics, each represents a scene of a typical irregular shape. Unlike most existing instance segmentation datasets of regular objects, iShape has many characteristics that challenge existing instance segmentation algorithms, such as large overlaps between bounding boxes of instances, extreme aspect ratios, and large numbers of connected components per instance. We benchmark popular instance segmentation methods on iShape and find their performance drop dramatically. Hence, we propose an affinity-based instance segmentation algorithm, called ASIS, as a stronger baseline. ASIS explicitly combines perception and reasoning to solve Arbitrary Shape Instance Segmentation including irregular objects. Experimental results show that ASIS outperforms the state-of-the-art on iShape. Dataset and code are available at https://ishape.github.io

Figures (9)

• Figure 1: A typical scene of objects with irregular shape and similar appearance. It has many characteristics that challenge instance segmentation algorithms, including the large overlaps between bounding boxes of objects, extreme aspect ratios (bounding box of the grey mask), and large numbers of connected components in one instance (green and blue masks).
• Figure 2: The six sub-datasets in iShape.
• Figure 3: Overview of ASIS. In the training stage, the network learns to predict the semantic segmentation as well as the affinity map. In the inference stage, first, build graph operation transforms the predicted affinity map into a sparse undirected graph by setting pixels as nodes and the affinity between pixels as edges. Then the graph merge algorithm is applied to the graph. The algorithm will cluster the pixels to yield class-agnostic instance segmentation. Finally, the class assign module will add a category with confidence to each instance using the result of semantic segmentation.
• Figure 4: Illustration for affinity kernels. (a) ASIS affinity kernel could connect these two segments with two neighbors (blue points). (b) GMIS affinity kernel cannot reach the right segment. (c) Examples of failure case for SSAP affinity kernel. For higher resolutions (top), $5 \times 5$ affinity window cannot reach the segment on the right. For lower resolutions (bottom), the view of thin antennas are lost in the resized feature maps.
• Figure 5: Two example false cases of ASIS on iShape-Antenna. (a) Two antennas merged into one (blue and orange). (b) ASIS fails to connect the right parts of an object (red and sky blue).