Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OMG-Seg: Is One Model Good Enough For All Segmentation?

Xiangtai Li, Haobo Yuan, Wei Li, Henghui Ding, Size Wu, Wenwei Zhang, Yining Li, Kai Chen, Chen Change Loy

TL;DR

It is shown that OMG-Seg, a transformer-based encoder-decoder architecture with task-specific queries and outputs, can support over ten distinct segmentation tasks and yet significantly reduce computational and parameter overhead across various tasks and datasets.

Abstract

In this work, we address various segmentation tasks, each traditionally tackled by distinct or partially unified models. We propose OMG-Seg, One Model that is Good enough to efficiently and effectively handle all the segmentation tasks, including image semantic, instance, and panoptic segmentation, as well as their video counterparts, open vocabulary settings, prompt-driven, interactive segmentation like SAM, and video object segmentation. To our knowledge, this is the first model to handle all these tasks in one model and achieve satisfactory performance. We show that OMG-Seg, a transformer-based encoder-decoder architecture with task-specific queries and outputs, can support over ten distinct segmentation tasks and yet significantly reduce computational and parameter overhead across various tasks and datasets. We rigorously evaluate the inter-task influences and correlations during co-training. Code and models are available at https://github.com/lxtGH/OMG-Seg.

OMG-Seg: Is One Model Good Enough For All Segmentation?

TL;DR

It is shown that OMG-Seg, a transformer-based encoder-decoder architecture with task-specific queries and outputs, can support over ten distinct segmentation tasks and yet significantly reduce computational and parameter overhead across various tasks and datasets.

Abstract

In this work, we address various segmentation tasks, each traditionally tackled by distinct or partially unified models. We propose OMG-Seg, One Model that is Good enough to efficiently and effectively handle all the segmentation tasks, including image semantic, instance, and panoptic segmentation, as well as their video counterparts, open vocabulary settings, prompt-driven, interactive segmentation like SAM, and video object segmentation. To our knowledge, this is the first model to handle all these tasks in one model and achieve satisfactory performance. We show that OMG-Seg, a transformer-based encoder-decoder architecture with task-specific queries and outputs, can support over ten distinct segmentation tasks and yet significantly reduce computational and parameter overhead across various tasks and datasets. We rigorously evaluate the inter-task influences and correlations during co-training. Code and models are available at https://github.com/lxtGH/OMG-Seg.
Paper Structure (14 sections, 4 figures, 10 tables)

This paper contains 14 sections, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Unified Task Representation
  5. OMG-Seg Architecture
  6. Training and Inference
  7. Experiments
  8. Main Results
  9. Ablation Study and Analysis
  10. Conclusion
  11. Appendix
  12. More Method Details
  13. More Experiment Results
  14. More Visualization Example

Figures (4)

  • Figure 1: OMG-Seg can handle over ten different segmentation tasks in one framework, including image-level and video-level segmentation tasks, interactive segmentation, and open-vocabulary segmentation. To our knowledge, this is the first model to unify these four directions.
  • Figure 2: OMG-Seg meta-architecture. (a) OMG-Seg follows the architecture of Mask2Former cheng2021mask2former, containing a backbone (CLIP Visual Encoder), a pixel decoder, and a mask decoder. The different parts are a shared mask decoder for both image and video segmentation and a visual prompt encoder. We use two types of mask queries, i.e., semantic queries, for instance/semantic masks or mask tubes, and location queries that encode box or point prompts. (b) One decoder layer in the Mask Decoder. The location queries skip the self-attention operation as they are only conditioned on the image content and the location prompts. (c) The forward pass of OMG-Seg in training and inference. We use CLIP's text encoder to represent category names and classify masks by calculating cosine similarity between mask features and text embeddings.
  • Figure 3: Functional Visualization of OMG-Seg model. We list five different tasks from four datasets as examples. Our method achieves high-quality segmentation, tracking, and as well as interactive segmentation in one shared model.
  • Figure 4: More functional Visualization of OMG-Seg model. In addition to five different tasks of the main paper, we also visualize the open-vocabulary segmentation results: open-vocabulary panoptic segmentation results on ADE-20k, open-vocabulary interactive segmentation results on ImageNet 1k dataset.