Image Segmentation in Foundation Model Era: A Survey

TL;DR

This survey addresses how foundation models reshape image segmentation by organizing FM-based methods into generic GIS and promptable PIS, and by analyzing emergence of segmentation knowledge in CLIP, diffusion models, and DINO. It categorizes approaches by semantic, instance, and panoptic tasks across CLIP-, DM-, DINO-, and SAM-based solutions, including training-free, zero-shot, few-shot, and interactive settings, and discusses how these FMs are composed to leverage their complementary strengths. The paper highlights over 300 approaches, identifies open questions such as explanation of emergent capabilities, object hallucination in MLLMs-based methods, and the need for scalable data engines and efficiency improvements, and provides a public resource to track developments. Overall, the work crystallizes current FM-driven segmentation trends and sketches practical paths toward robust, flexible, and scalable segmentation systems in real-world scenarios.

Abstract

Image segmentation is a long-standing challenge in computer vision, studied continuously over several decades, as evidenced by seminal algorithms such as N-Cut, FCN, and MaskFormer. With the advent of foundation models (FMs), contemporary segmentation methodologies have embarked on a new epoch by either adapting FMs (e.g., CLIP, Stable Diffusion, DINO) for image segmentation or developing dedicated segmentation foundation models (e.g., SAM). These approaches not only deliver superior segmentation performance, but also herald newfound segmentation capabilities previously unseen in deep learning context. However, current research in image segmentation lacks a detailed analysis of distinct characteristics, challenges, and solutions associated with these advancements. This survey seeks to fill this gap by providing a thorough review of cutting-edge research centered around FM-driven image segmentation. We investigate two basic lines of research -- generic image segmentation (i.e., semantic segmentation, instance segmentation, panoptic segmentation), and promptable image segmentation (i.e., interactive segmentation, referring segmentation, few-shot segmentation) -- by delineating their respective task settings, background concepts, and key challenges. Furthermore, we provide insights into the emergence of segmentation knowledge from FMs like CLIP, Stable Diffusion, and DINO. An exhaustive overview of over 300 segmentation approaches is provided to encapsulate the breadth of current research efforts. Subsequently, we engage in a discussion of open issues and potential avenues for future research. We envisage that this fresh, comprehensive, and systematic survey catalyzes the evolution of advanced image segmentation systems. A public website is created to continuously track developments in this fast advancing field: \url{https://github.com/stanley-313/ImageSegFM-Survey}.

Figures (5)

• Figure 1: Image segmentation tasks reviewed in this survey. Generic image segmentation: (a) semantic segmentation, (b) instance segmentation, (c) panoptic segmentation; Promptable image segmentation: (d) interactive segmentation, (e) referring segmentation, (f) few-shot segmentation.
• Figure 2: Overview of this survey.
• Figure 3: (a) Illustrations of how segmentation derived from FMs. Briefly speaking, Modifying CLIP's attention pooling to location-aware attentions can obtain segmentation features. Merging cross-attention maps and self-attention maps in DMs can produce precise semantic segments. DINO naturally contains segmentation properties in the last attention maps of the class token. (b) shows some visualization examples.
• Figure 4: Overview of Foundation Model based GIS (§\ref{['sec:4']}).
• Figure 5: Overview of Foundation Model based PIS (§\ref{['sec:5']}).