Panoptic Perception: A Novel Task and Fine-grained Dataset for Universal Remote Sensing Image Interpretation

TL;DR

The paper tackles the fragmentation of remote sensing image interpretation by introducing Panoptic Perception, a unified task that combines pixel-level, instance-level, and image-level understanding. It presents FineGrip, a fine-grained RSI dataset tailored for multi-task learning, including 20 foreground aircraft categories, 5 background classes, and 13,245 captions, along with a SAM-based semi-automatic annotation workflow. An end-to-end, multi-task baseline integrates a pixel-level module, a panoptic segmentation module, and a caption module, optimized jointly via $\mathcal{L}_{total} = \mathcal{L}_{seg} + \lambda \mathcal{L}_{cap}$. Experiments on FineGrip show that cross-task interaction improves both segmentation quality (PQ, SQ, RQ) and caption quality (BLEU), validating the feasibility and value of unified panoptic perception for RSIs.

Abstract

Current remote-sensing interpretation models often focus on a single task such as detection, segmentation, or caption. However, the task-specific designed models are unattainable to achieve the comprehensive multi-level interpretation of images. The field also lacks support for multi-task joint interpretation datasets. In this paper, we propose Panoptic Perception, a novel task and a new fine-grained dataset (FineGrip) to achieve a more thorough and universal interpretation for RSIs. The new task, 1) integrates pixel-level, instance-level, and image-level information for universal image perception, 2) captures image information from coarse to fine granularity, achieving deeper scene understanding and description, and 3) enables various independent tasks to complement and enhance each other through multi-task learning. By emphasizing multi-task interactions and the consistency of perception results, this task enables the simultaneous processing of fine-grained foreground instance segmentation, background semantic segmentation, and global fine-grained image captioning. Concretely, the FineGrip dataset includes 2,649 remote sensing images, 12,054 fine-grained instance segmentation masks belonging to 20 foreground things categories, 7,599 background semantic masks for 5 stuff classes and 13,245 captioning sentences. Furthermore, we propose a joint optimization-based panoptic perception model. Experimental results on FineGrip demonstrate the feasibility of the panoptic perception task and the beneficial effect of multi-task joint optimization on individual tasks. The dataset will be publicly available.

Figures (9)

• Figure 1: The proposed fine-grained Panoptic Perception task. (a) The definition and scope of panoptic perception Task. (b) Qualitative comparison of panoptic perception and other interpretation tasks.
• Figure 2: Number of per-category masks in the FineGrip dataset across training and validation sets.
• Figure 3: Examples of Annotations in FineGrip. (a) Foreground fine-grained instance segmentation annotations.(b) Background stuff semantic segmentation annotations.
• Figure 4: Examples of Per-Image Annotations from FineGrip. In the captions, red highlights detail the fine-grained categories and quantities of aircraft instances, while blue highlights indicate relationships between instances.
• Figure 5: Examples of Per-Image Complete Annotations from FineGrip. In the captions, red highlights detail the fine-grained categories and quantities of aircraft instances, while blue highlights indicate relationships between instances.