Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Transformer-Based Visual Segmentation: A Survey

Xiangtai Li, Henghui Ding, Haobo Yuan, Wenwei Zhang, Jiangmiao Pang, Guangliang Cheng, Kai Chen, Ziwei Liu, Chen Change Loy

TL;DR

This survey consolidates transformer-based visual segmentation by detailing a DETR-inspired meta-architecture (backbone, object queries, transformer decoder) and a five-fold method taxonomy, covering strong representations, decoder cross-attention, object-query optimization, query-based association, and conditional query fusion. It catalogs datasets, metrics, and representative works across 2D/3D, image/video, and medical domains, and includes re-benchmarking to enable fair comparisons under unified settings. The paper also surveys five related subfields (point cloud, foundation-model tuning/open vocabulary, domain adaptation, efficiency, and class-agnostic tracking) and highlights practical trends such as unified image–video segmentation, open vocabulary approaches, and long-term memory for video. By identifying open challenges and proposing concrete directions, it aims to guide researchers and practitioners toward robust, scalable, and generalizable transformer-based segmentation systems with broad real-world impact.

Abstract

Visual segmentation seeks to partition images, video frames, or point clouds into multiple segments or groups. This technique has numerous real-world applications, such as autonomous driving, image editing, robot sensing, and medical analysis. Over the past decade, deep learning-based methods have made remarkable strides in this area. Recently, transformers, a type of neural network based on self-attention originally designed for natural language processing, have considerably surpassed previous convolutional or recurrent approaches in various vision processing tasks. Specifically, vision transformers offer robust, unified, and even simpler solutions for various segmentation tasks. This survey provides a thorough overview of transformer-based visual segmentation, summarizing recent advancements. We first review the background, encompassing problem definitions, datasets, and prior convolutional methods. Next, we summarize a meta-architecture that unifies all recent transformer-based approaches. Based on this meta-architecture, we examine various method designs, including modifications to the meta-architecture and associated applications. We also present several closely related settings, including 3D point cloud segmentation, foundation model tuning, domain-aware segmentation, efficient segmentation, and medical segmentation. Additionally, we compile and re-evaluate the reviewed methods on several well-established datasets. Finally, we identify open challenges in this field and propose directions for future research. The project page can be found at https://github.com/lxtGH/Awesome-Segmentation-With-Transformer. We will also continually monitor developments in this rapidly evolving field.

Transformer-Based Visual Segmentation: A Survey

TL;DR

This survey consolidates transformer-based visual segmentation by detailing a DETR-inspired meta-architecture (backbone, object queries, transformer decoder) and a five-fold method taxonomy, covering strong representations, decoder cross-attention, object-query optimization, query-based association, and conditional query fusion. It catalogs datasets, metrics, and representative works across 2D/3D, image/video, and medical domains, and includes re-benchmarking to enable fair comparisons under unified settings. The paper also surveys five related subfields (point cloud, foundation-model tuning/open vocabulary, domain adaptation, efficiency, and class-agnostic tracking) and highlights practical trends such as unified image–video segmentation, open vocabulary approaches, and long-term memory for video. By identifying open challenges and proposing concrete directions, it aims to guide researchers and practitioners toward robust, scalable, and generalizable transformer-based segmentation systems with broad real-world impact.

Abstract

Visual segmentation seeks to partition images, video frames, or point clouds into multiple segments or groups. This technique has numerous real-world applications, such as autonomous driving, image editing, robot sensing, and medical analysis. Over the past decade, deep learning-based methods have made remarkable strides in this area. Recently, transformers, a type of neural network based on self-attention originally designed for natural language processing, have considerably surpassed previous convolutional or recurrent approaches in various vision processing tasks. Specifically, vision transformers offer robust, unified, and even simpler solutions for various segmentation tasks. This survey provides a thorough overview of transformer-based visual segmentation, summarizing recent advancements. We first review the background, encompassing problem definitions, datasets, and prior convolutional methods. Next, we summarize a meta-architecture that unifies all recent transformer-based approaches. Based on this meta-architecture, we examine various method designs, including modifications to the meta-architecture and associated applications. We also present several closely related settings, including 3D point cloud segmentation, foundation model tuning, domain-aware segmentation, efficient segmentation, and medical segmentation. Additionally, we compile and re-evaluate the reviewed methods on several well-established datasets. Finally, we identify open challenges in this field and propose directions for future research. The project page can be found at https://github.com/lxtGH/Awesome-Segmentation-With-Transformer. We will also continually monitor developments in this rapidly evolving field.
Paper Structure (32 sections, 4 equations, 4 figures, 25 tables)

This paper contains 32 sections, 4 equations, 4 figures, 25 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Problem Definition
  4. Datasets and Metrics
  5. Segmentation Approaches Before Transformer
  6. Transformer Basics
  7. Methods: A Survey
  8. Meta-Architecture
  9. Method Categorization
  10. Strong Representations
  11. Cross-Attention Design in Decoder
  12. Optimizing Object Query
  13. Using Query For Association
  14. Conditional Query Fusion
  15. Specific Subfields
  16. ...and 17 more sections

Figures (4)

  • Figure 1: A diagram that summarizes this survey. Different colors represent specific sections. Best viewed in color.
  • Figure 2: Illustration of different segmentation tasks. The examples are sampled from the VIP-Seg dataset miao2022large. For (V)SS, the same color indicates the same class. For (V)IS and (V)PS, different instances are represented by different colors.
  • Figure 3: Illustration of (a) meta-architecture and (b) common operations in the decoder.
  • Figure 4: Illustration of object query in video segmentation.