CoMBO: Conflict Mitigation via Branched Optimization for Class Incremental Segmentation

TL;DR

CoMBO tackles Class Incremental Segmentation by introducing branched optimization to decouple learning of new classes from distillation of old ones. The Query Conflict Reduction module refines new-class queries via class-specific adapters, while Half-Distillation-Half-Learning (HDHL) and Importance-Based Knowledge Distillation (IKD) balance stability and plasticity through targeted distillation on logits and features. Extensive ADE20K experiments show state-of-the-art gains in both Class Incremental Panoptic Segmentation and Class Incremental Semantic Segmentation, with notable improvements for incremental classes and efficient parameter overhead. The approach demonstrates that separating conflicting objectives and weighting essential knowledge can substantially mitigate forgetting while enabling rapid integration of novel categories.

Abstract

Effective Class Incremental Segmentation (CIS) requires simultaneously mitigating catastrophic forgetting and ensuring sufficient plasticity to integrate new classes. The inherent conflict above often leads to a back-and-forth, which turns the objective into finding the balance between the performance of previous~(old) and incremental~(new) classes. To address this conflict, we introduce a novel approach, Conflict Mitigation via Branched Optimization~(CoMBO). Within this approach, we present the Query Conflict Reduction module, designed to explicitly refine queries for new classes through lightweight, class-specific adapters. This module provides an additional branch for the acquisition of new classes while preserving the original queries for distillation. Moreover, we develop two strategies to further mitigate the conflict following the branched structure, \textit{i.e.}, the Half-Learning Half-Distillation~(HDHL) over classification probabilities, and the Importance-Based Knowledge Distillation~(IKD) over query features. HDHL selectively engages in learning for classification probabilities of queries that match the ground truth of new classes, while aligning unmatched ones to the corresponding old probabilities, thus ensuring retention of old knowledge while absorbing new classes via learning negative samples. Meanwhile, IKD assesses the importance of queries based on their matching degree to old classes, prioritizing the distillation of important features and allowing less critical features to evolve. Extensive experiments in Class Incremental Panoptic and Semantic Segmentation settings have demonstrated the superior performance of CoMBO. Project page: https://guangyu-ryan.github.io/CoMBO.

Figures (8)

• Figure 1: Comparison of our improved distillation and learning strategy (bottom) with previous conflicting strategy (top). Previous strategies impose contradictory supervision on the same target to find a balance, whereas our strategy, including the importance factor for distillation and the QCR module for adaptive learning, enables more compatible, target-specific supervision. The visualization on incremental classes ('Traffic Light') highlights the effectiveness of our strategy.
• Figure 2: Illustration of the Query Conflict Reduction (QCR) module. This module refines queries that predict incremental classes, allowing the processes of learning new features and retaining old features to occur separately. Note that the QCR module for previous incremental classes is frozen.
• Figure 3: Details of the Half-Distillation-Half-Learning strategy on $N$ queries, including classification loss on matched queries and Kullback-Leibler Divergence loss on the other queries. The latter involves the logits of both old and current classes.
• Figure 4: Qualitative results of CoMBO comparing to Baseline, Baseline+HDHL on 100-10 CIPS task of ADE20K.
• Figure 5: Ablation study of $r$ in QCR module, including PQ performance (left) and number of parameters (right).