Open Vocabulary Panoptic Segmentation With Retrieval Augmentation

TL;DR

RetCLIP addresses open vocabulary panoptic segmentation by integrating retrieval-augmented classification with CLIP-based scoring. It constructs a masked-segment feature database from paired image–text data and uses the input's masked features as queries to retrieve similar targets, mitigating domain shift between masked and natural image features. The approach fuses retrieval-derived scores with CLIP predictions and, when available, COCO-based fine-tuning, achieving notable gains on ADE20k (e.g., PQ up to 0.309) and demonstrating training-free and cross-dataset robustness. By enabling new classes through database updates rather than retraining, RetCLIP offers a scalable path for open vocabulary segmentation, while underscoring the importance of mask proposal quality.

Abstract

Given an input image and set of class names, panoptic segmentation aims to label each pixel in an image with class labels and instance labels. In comparison, Open Vocabulary Panoptic Segmentation aims to facilitate the segmentation of arbitrary classes according to user input. The challenge is that a panoptic segmentation system trained on a particular dataset typically does not generalize well to unseen classes beyond the training data. In this work, we propose RetCLIP, a retrieval-augmented panoptic segmentation method that improves the performance of unseen classes. In particular, we construct a masked segment feature database using paired image-text data. At inference time, we use masked segment features from the input image as query keys to retrieve similar features and associated class labels from the database. Classification scores for the masked segment are assigned based on the similarity between query features and retrieved features. The retrieval-based classification scores are combined with CLIP-based scores to produce the final output. We incorporate our solution with a previous SOTA method (FC-CLIP). When trained on COCO, the proposed method demonstrates 30.9 PQ, 19.3 mAP, 44.0 mIoU on the ADE20k dataset, achieving +4.5 PQ, +2.5 mAP, +10.0 mIoU absolute improvement over the baseline.

Figures (2)

• Figure 1: Overview of feature database construction
• Figure 2: Overview of RetCLIP. All components of the retrieval pipeline are shown in purple. The trainable components for cross-dataset fine-tuning are shown in green.