MISS: Memory-efficient Instance Segmentation Framework By Visual Inductive Priors Flow Propagation

TL;DR

The paper tackles data-scarce, resource-constrained instance segmentation by proposing MISS, a memory-efficient framework that injects visual inductive priors into preprocessing, augmentation, training, and inference. The pipeline leverages court geometry via Canny-Hough detection, identity-aware style transformations, and location-aware copy-paste augmentation to reduce data and compute requirements while preserving accuracy on the Synergy-basketball dataset, achieving substantial memory savings with competitive metrics. Through ablations and testing-time augmentations, MISS demonstrates that priors can replace heavy pretraining and large models in constrained settings, notably reducing memory usage (approximately 42% of a state-of-the-art baseline) yet maintaining strong performance. The approach holds promise for broader applicability in domains with strong background rules or priors, enabling efficient high-resolution, fine-grained segmentation without extensive computational resources.

Abstract

Instance segmentation, a cornerstone task in computer vision, has wide-ranging applications in diverse industries. The advent of deep learning and artificial intelligence has underscored the criticality of training effective models, particularly in data-scarce scenarios - a concern that resonates in both academic and industrial circles. A significant impediment in this domain is the resource-intensive nature of procuring high-quality, annotated data for instance segmentation, a hurdle that amplifies the challenge of developing robust models under resource constraints. In this context, the strategic integration of a visual prior into the training dataset emerges as a potential solution to enhance congruity with the testing data distribution, consequently reducing the dependency on computational resources and the need for highly complex models. However, effectively embedding a visual prior into the learning process remains a complex endeavor. Addressing this challenge, we introduce the MISS (Memory-efficient Instance Segmentation System) framework. MISS leverages visual inductive prior flow propagation, integrating intrinsic prior knowledge from the Synergy-basketball dataset at various stages: data preprocessing, augmentation, training, and inference. Our empirical evaluations underscore the efficacy of MISS, demonstrating commendable performance in scenarios characterized by limited data availability and memory constraints.

Figures (6)

• Figure 1: Comparison of performance and computational resource requirements between our approach and previous benchmark 2021sota2022sota, in Synergy-basketball dataset. Compared with other methods, our method significantly reduces the demand for computational resources. In the figure, the size of the circles represents the memory usage for each method.
• Figure 2: The overall of proposed instance segmentation framework. The visual inductive prior is fully utilized at each stage to make effective optimizations. This approach not only reduces computational resource consumption but also maintains solid model performance. We begin by employing the Canny-Hough operator to adaptively combine image-level prior to detect the basketball court's position. Subsequently, we leverage class-level prior for identity identification. We then utilize this information for style transformation of various objects, integrating image-level prior knowledge through copy-paste augmentation. Finally, model inference solely based on the detected basketball court's location.
• Figure 3: The illustrations for the cropping algorithm and location-based copy-paste augmentation. The top-left figure is the original image. The top-right one is cropped, with red lines detected by the Canny-Hough operator. The blue line shows a boundary based on image size, while the green lines indicate dynamic boundary from the detected lines. The two picture below display a region identified based on the maximum convex hull, which is determined using the endpoints of all lines detected by the Canny-Hough operator. The object marked by a dotted line is pasted . The subclass attributes of the object are determined by its bounding box coordinates.
• Figure 4: The results visualization of proposed method. (a) is the result of using simple copy-paste augmentation. (b) showcases the results using our proposed method. During the experiments, to ensure the simplicity of the methods for easy comparison, we excluded any post-processing or testing-time augmentation mentioned in this paper. (c) and (d) provide a magnified comparison of the areas in the images where artifacts are generated.
• Figure 5: The demo of identity-based style transfer applied to basketball players. Significant variations in appearance are evident after the hue or RGB transformation. In the left example, there is a noticeable change in skin tone, while in the right example, the player's jersey changes dramatically, almost as if he has switched to a different team.