SimCS: Simulation for Domain Incremental Online Continual Segmentation

TL;DR

This work tackles Online Domain-Incremental Continual Segmentation (ODICS), a realistic setting where a segmentation model learns from a continuous stream of labeled images across changing domains with limited compute and no explicit task boundaries. It introduces SimCS, a parameter-free approach that regularizes continual learning by incorporating on-the-fly simulated data $S_{\text{sim}_t}$ aligned to the real task, without storing past data. SimCS is shown to consistently improve a range of continual learning strategies across a 4-domain autonomous-driving segmentation benchmark, with robustness to simulators (CARLA and VIPER), beneficial pretraining effects, and favorable forward and backward transfer. The results suggest SimCS as a practical, privacy-preserving, and computation-aware regularizer for online domain-incremental semantic segmentation, with clear potential for real-world continual updates in driving systems.

Abstract

Continual Learning is a step towards lifelong intelligence where models continuously learn from recently collected data without forgetting previous knowledge. Existing continual learning approaches mostly focus on image classification in the class-incremental setup with clear task boundaries and unlimited computational budget. This work explores the problem of Online Domain-Incremental Continual Segmentation (ODICS), where the model is continually trained over batches of densely labeled images from different domains, with limited computation and no information about the task boundaries. ODICS arises in many practical applications. In autonomous driving, this may correspond to the realistic scenario of training a segmentation model over time on a sequence of cities. We analyze several existing continual learning methods and show that they perform poorly in this setting despite working well in class-incremental segmentation. We propose SimCS, a parameter-free method complementary to existing ones that uses simulated data to regularize continual learning. Experiments show that SimCS provides consistent improvements when combined with different CL methods.

Figures (5)

• Figure 1: Online Domain Incremental Continual Segmentation (ODICS) with Simulated Data (SimCS). At each time step $t$, ODICS reveals a batch of labeled images $S_t$ with size $B_t$ from a certain domain, where different domains are presented sequentially to the model. SimCS generates a batch with size $B_t$ of simulated data $S_{\text{sim}_t}$ on the fly and aligns its label space to the real stream. The concatenated batch of real and simulated data is presented to the model to aid continual training and to mitigate forgetting previously learnt domains.
• Figure 2: Effect of Varying Sim-Real Ratio on the Performance Gain. We analyze the effect of varying the ratio between simulation and real data from {1/4, 1/2, 1, 2, 4, 5, 8, 10} on the performance gain for each observed domain. We find that SimCS provides notable performance improvement on a wide range of ratios ($\leq$ 5).
• Figure 3: Comparison under different computational budgets. We allow NT, NT+CARLA, and NT+VIPER different computational budgets for training on each received batch from the stream, measured by the number of training iterations. We measure the performance on each observed domain when varying the budget to {1, 2, 3, 4,6,8,10} training iterations.
• Figure 4: Forward and backward transfer under different domain orders. We analyze the forward and backward transfer during ODICS of both NT and NT+VIPER under different domain orders. The x-axis represents the observed domain within the stream while the y-axis shows the domain, on which we are evaluating the model. SimCS with VIPER improves both the forward (lower triangular) and backward (upper triangular) transfer in ODICS under different domain orders.
• Figure 5: Online Data Incremental Continual Semantic Segmentation with SimCS. The learner receives a mixed batch from different domains. SimCS provides simulated data with aligned labels space to the real data. Note that different domains might present the learner with different amounts of data.