A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application

TL;DR

This survey targets continual semantic segmentation (CSS), a dense-prediction continual learning problem challenged by catastrophic forgetting and semantic drift. It articulates problem formulations, taxonomies, and benchmarks, and organizes existing work into data-replay and data-free families, with extensions into domain-, class-, and modality-incremental settings. The authors provide unified qualitative and quantitative analyses, present a CSS benchmark and reproducible insights, and discuss applications in autonomous driving, remote sensing, and medical imaging, along with future directions such as brain-inspired models and foundation-model driven approaches. Overall, the work clarifies the landscape of CSS methods, compares performance across benchmarks, and outlines practical paths toward robust, scalable open-world segmentation in real systems.

Abstract

Continual learning, also known as incremental learning or life-long learning, stands at the forefront of deep learning and AI systems. It breaks through the obstacle of one-way training on close sets and enables continuous adaptive learning on open-set conditions. In the recent decade, continual learning has been explored and applied in multiple fields especially in computer vision covering classification, detection and segmentation tasks. Continual semantic segmentation (CSS), of which the dense prediction peculiarity makes it a challenging, intricate and burgeoning task. In this paper, we present a review of CSS, committing to building a comprehensive survey on problem formulations, primary challenges, universal datasets, neoteric theories and multifarious applications. Concretely, we begin by elucidating the problem definitions and primary challenges. Based on an in-depth investigation of relevant approaches, we sort out and categorize current CSS models into two main branches including data-replay and data-free sets. In each branch, the corresponding approaches are similarity-based clustered and thoroughly analyzed, following qualitative comparison and quantitative reproductions on relevant datasets. Besides, we also introduce four CSS specialities with diverse application scenarios and development tendencies. Furthermore, we develop a benchmark for CSS encompassing representative references, evaluation results and reproductions, which is available at~\url{https://github.com/YBIO/SurveyCSS}. We hope this survey can serve as a reference-worthy and stimulating contribution to the advancement of the life-long learning field, while also providing valuable perspectives for related fields.

Figures (11)

• Figure 1: Illustration of catastrophic forgetting and semantic drift in continual semantic segmentation. (a): The decision boundary varies as new data involves, which normally encounters classifier failure. (b): The manifestation of catastrophic forgetting and semantic drift in CSS, leading to semantic confusion and model degradation reflected in the predicted results.
• Figure 2: The roadmap of CSS. The representative methods are categorized chronologically. Please note that these methods are not committed to covering all CSS methods but are simply used to validate the taxonomy. Refer to the main text for a more comprehensive summary.
• Figure 3: An elaborated taxonomy of continual semantic segmentation methods.
• Figure 4: The flowcharts of different CSS specialities.
• Figure 5: Concerning the dependence on the old data, there are generally data-replay and data-free CSS methods. According to the dependence on the incremental data, data-free branch covers unlimited, few-shot and zero-shot approaches.