High Quality Segmentation for Ultra High-resolution Images

TL;DR

This work tackles the challenge of segmenting ultra high-resolution images (4K–6K) where traditional downsampling or cascade-based refinements are costly or fail to preserve details. It introduces the Continuous Refinement Model (CRM), which pairs a Continuous Alignment Module (CAM) with a pixel-wise implicit function to refine coarse masks without explicit upsampling, and employs a multi-resolution inference strategy to bridge the training/testing resolution gap. The key contributions are the CAM design for continuous feature alignment, a LIIF-inspired implicit function for pixel-wise refinement with neighborhood aggregation, and a training/inference framework that enables fast, high-quality refinement on ultra high-resolution data. Empirical results on the BIG dataset and related benchmarks show that CRM outperforms state-of-the-art refinement methods in both accuracy (IoU/mBA) and speed, and can generalize to support panoptic and entity segmentation without extra finetuning.

Abstract

To segment 4K or 6K ultra high-resolution images needs extra computation consideration in image segmentation. Common strategies, such as down-sampling, patch cropping, and cascade model, cannot address well the balance issue between accuracy and computation cost. Motivated by the fact that humans distinguish among objects continuously from coarse to precise levels, we propose the Continuous Refinement Model~(CRM) for the ultra high-resolution segmentation refinement task. CRM continuously aligns the feature map with the refinement target and aggregates features to reconstruct these images' details. Besides, our CRM shows its significant generalization ability to fill the resolution gap between low-resolution training images and ultra high-resolution testing ones. We present quantitative performance evaluation and visualization to show that our proposed method is fast and effective on image segmentation refinement. Code will be released at https://github.com/dvlab-research/Entity.

Figures (7)

• Figure 1: Coarse mask refinement results. (a) Coarse mask from PSP zhao2017pyramid, (b) refined mask of state-of-the-art cheng2020cascadepsp, and (c) refined mask of our proposed CRM. The image is from BIG (2K$\sim$6K res).
• Figure 2: Structure difference between (a) Cascade-based decoder in model cheng2020cascadepsp and (b) our CRM. We can see CRM is much simpler, which is the base of our speed advantage.
• Figure 3: The general framework of CRM. The upper part is the structure of the model. The lower part is the training and testing process of CRM. From the lower part, we can also see the resolution gap between low-resolution training and high-resolution testing.
• Figure 4: Visualization of refinement steps in our inference strategy. From left to right, top to down: $M_{\text{coarse}}$, refined mask $M^{i}_{\text{refined}}, i\in \{1,2,3,4\}$ (The rescale ratios are 0.125, 0.25, 0.5, and 1.0 here.), and overlay $M^{4}_{\text{refined}}$ on the original image.
• Figure 5: Qualitative comparison between Segfix, CascadePSP and CRM on the coarse mask from FCN, DeepLabV3+, RefineNet and PSPNet. The images are from BIG (2K $\sim$ 6K). And the black-white mask in bottom left part of first column is the coarse mask.